Sep 19 15:10:08.989923 ip-10-0-141-0 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 19 15:10:08.989936 ip-10-0-141-0 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 19 15:10:08.989944 ip-10-0-141-0 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 19 15:10:08.990220 ip-10-0-141-0 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 19 15:10:18.307159 ip-10-0-141-0 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 19 15:10:18.307172 ip-10-0-141-0 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a262bda731ec4efaa11a5ce60d487507 --
Sep 19 15:12:16.692921 ip-10-0-141-0 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 15:12:19.973357 ip-10-0-141-0 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 15:12:19.973357 ip-10-0-141-0 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 15:12:19.973357 ip-10-0-141-0 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 15:12:19.973357 ip-10-0-141-0 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 15:12:19.973357 ip-10-0-141-0 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 15:12:19.974161 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.973451    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 15:12:19.977266 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977245    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 15:12:19.977266 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977263    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 15:12:19.977266 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977268    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 15:12:19.977266 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977272    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977277    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977281    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977285    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977288    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977292    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977296    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977299    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977302    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977306    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977310    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977313    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977317    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977321    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977325    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977329    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977340    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977344    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977347    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 15:12:19.977514 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977351    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977355    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977359    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977363    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977366    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977370    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977374    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977378    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977381    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977385    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977389    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977393    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977398    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977402    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977408    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977414    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977418    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977422    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977427    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977430    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977434    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 15:12:19.978290 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977439    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977443    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977447    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977451    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977455    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977459    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977463    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977467    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977471    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977477    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977484    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977488    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977493    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977496    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977500    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977507    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977512    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977517    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977522    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 15:12:19.979181 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977527    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977532    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977536    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977541    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977545    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977549    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977553    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977559    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977564    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977568    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977572    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977576    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977580    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977585    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977589    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977593    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977616    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977621    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977625    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977629    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 15:12:19.979973 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977633    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977637    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977640    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977647    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977652    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977656    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977661    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977664    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977668    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977673    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977676    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977680    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977684    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977688    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977692    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977696    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.977700    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978807    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978816    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 15:12:19.980818 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978822    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978828    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978833    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978836    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978841    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978845    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978849    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978855    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978859    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978863    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978867    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978871    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978875    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978882    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978887    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978891    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978896    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978900    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978905    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978909    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 15:12:19.981672 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978913    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978918    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978922    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978926    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978930    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978934    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978938    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978943    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978946    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978950    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978954    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978959    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978964    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978968    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978972    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978976    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978980    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978984    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978988    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.978994    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 15:12:19.982286 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979000    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979004    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979008    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979012    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979018    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979022    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979027    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979031    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979035    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979039    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979043    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979047    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979051    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979054    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979058    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979062    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979066    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979070    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979074    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979078    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 15:12:19.982862 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979082    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979086    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979089    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979093    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979098    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979102    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979106    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979119    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979123    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979127    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979131    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979137    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979140    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979145    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979149    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979154    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979158    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979161    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979166    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979170    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 15:12:19.983406 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979174    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979177    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979181    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979184    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979188    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979192    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979196    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979200    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979203    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979207    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979211    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979216    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979223    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979227    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979231    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979235    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979241    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979818    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979826    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 15:12:19.984121 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979831    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979835    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979839    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979843    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979847    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979851    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979857    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979861    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979865    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979869    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979873    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979877    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979881    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979885    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979889    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979893    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979897    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979901    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979905    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979909    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 15:12:19.984577 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979913    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979918    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979922    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979926    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979930    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979934    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979939    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979942    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979946    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979951    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979955    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979959    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979963    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979967    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979971    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979975    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979979    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979983    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979987    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 15:12:19.985138 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979991    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979995    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.979999    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980003    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980008    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980013    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980017    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980021    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980025    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980029    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980033    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980037    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980042    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980046    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980050    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980054    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980058    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980062    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980067    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 15:12:19.985626 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980071    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980075    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980080    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980085    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980091    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980096    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980100    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980104    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980108    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980112    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980116    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980119    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980128    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980132    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980137    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980144    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980149    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980154    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980158    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 15:12:19.986087 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980163    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980167    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980172    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980175    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980180    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980186    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980191    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980196    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980200    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980204    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980209    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980213    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980217    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980221    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980226    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980229    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980234    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980238    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980241    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.980245    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 15:12:19.986533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980348    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980392    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980403    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980410    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980416    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980423    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980430    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980436    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980447    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980452    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980457    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980462    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980466    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980471    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980475    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980480    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980485    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980489    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980495    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980499    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980504    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980508    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980513    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980519    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 15:12:19.987093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980523    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980528    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980534    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980539    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980544    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980548    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980553    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980557    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980563    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980568    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980572    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980577    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980581    2579 flags.go:64] FLAG: --enable-server="true"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980588    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980622    2579 flags.go:64] FLAG: --event-burst="100"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980628    2579 flags.go:64] FLAG: --event-qps="50"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980632    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980638    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980642    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980653    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980657    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980662    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980666    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980671    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980676    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 15:12:19.987686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980681    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980685    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980690    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980694    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980698    2579 flags.go:64] FLAG: --feature-gates=""
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980704    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980709    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980713    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980718    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980723    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980727    2579 flags.go:64] FLAG: --help="false"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980734    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-0.ec2.internal"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980738    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980743    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980748    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980753    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980759    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980763    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980767    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980772    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980777    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980782    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980787    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980791    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 19 15:12:19.988341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980796    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980801    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980806    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980810    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980814    2579 flags.go:64] FLAG: --lock-file=""
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980818    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980822    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980827    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980834    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980839    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980843    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980847    2579 flags.go:64] FLAG: --logging-format="text"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980851    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980857    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980861    2579 flags.go:64] FLAG: --manifest-url=""
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980866    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980872    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980877    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980883    2579 flags.go:64] FLAG: --max-pods="110"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980889    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980893    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980898    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980902    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980907    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980912    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 15:12:19.988938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980916    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980927    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980931    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980938    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980943    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980947    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980956    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980961    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980965    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980970    2579 flags.go:64] FLAG: --port="10250"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980975    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980979    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05750eaa453998b02"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980984    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980989    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980993    2579 flags.go:64] FLAG: --register-node="true"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.980998    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981009    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981014    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981018    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981022    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981028    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981033    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981037    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981041    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981046    2579 flags.go:64] FLAG: --runonce="false"
Sep 19 15:12:19.989533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981050    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981055    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981061    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981066    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981081    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981086    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981090    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981095    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981100    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981105    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981111    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981115    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981120    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981124    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981129    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981138    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981142    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981148    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981153    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981158    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981162    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981166    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981171    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981176    2579 flags.go:64] FLAG: --v="2"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981182    2579 flags.go:64] FLAG: --version="false"
Sep 19 15:12:19.990146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981188    2579 flags.go:64] FLAG: --vmodule=""
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981197    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981202    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981371    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981379    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981385    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981389    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981393    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981398    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981404    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981408    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981413    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981417    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981421    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981425    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981429    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981433    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981439    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981443    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981446    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 15:12:19.990768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981450    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981456    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981462    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981467    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981471    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981483    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981488    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981492    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981496    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981500    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981504    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981508    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981512    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981516    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981520    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981524    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981528    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981532    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981535    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 15:12:19.991250 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981539    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981543    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981549    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981554    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981565    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981569    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981574    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981580    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981585    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981590    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981595    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981618    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981622    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981626    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981630    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981634    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981637    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981641    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981645    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981650    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 15:12:19.991768 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981655    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981659    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981663    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981667    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981670    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981674    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981678    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981681    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981685    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981689    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981693    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981696    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981700    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981704    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981710    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981714    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981717    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981721    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981726    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981730    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 15:12:19.992247 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981736    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981741    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981747    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981751    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981755    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981760    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981764    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981768    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981772    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981776    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981780    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981784    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981789    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981793    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981797    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981802    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981806    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981810    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981814    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981818    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 15:12:19.992735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981822    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981826    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.981829    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.981837    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.989499    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.989514    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.989532    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.989538    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989591    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989596    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989618    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989623    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989626    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989629    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989632    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 15:12:19.993206 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989634    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989637    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989640    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989642    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989645    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989647    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989650    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989653    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989655    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989658    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989660    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989663    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989665    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989668    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989670    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989673    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989675    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989677    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989680    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989682    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 15:12:19.993579 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989685    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989687    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989691    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989694    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989697    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989699    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989702    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989704    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989707    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989709    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989711    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989714    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989716    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989719    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989721    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989724    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989726    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989728    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989731    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989733    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 15:12:19.994086 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989736    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989738    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989754    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989757    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989760    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989768    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989771    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989773    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989776    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989778    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989781    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989784    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989786    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989788    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989791    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989795    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989797    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989801    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989805    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 15:12:19.994883 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989808    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989812    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989815    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989818    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989822    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989824    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989827    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989829    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989832    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989834    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989836    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989839    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989841    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989844    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989846    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989849    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989851    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989853    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989856    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 15:12:19.995585 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989859    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989861    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989864    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989866    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989869    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989871    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989873    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989876    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989878    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989881    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989884    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989888    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989891    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.989894    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.989899    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 15:12:19.996063 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990008    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990013    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990016    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990019    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990021    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990024    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990027    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990029    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990032    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990035    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990037    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990040    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990043    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990045    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990048    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990050    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990053    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990056    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990058    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990061    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 15:12:19.996432 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990063    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990065    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990068    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990070    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990073    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990075    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990078    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990080    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990083    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990085    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990088    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990090    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990093    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990095    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990097    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990099    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990102    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990104    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990107    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990109    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 15:12:19.996915 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990112    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990115    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990120    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990124    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990127    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990130    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990133    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990136    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990138    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990141    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990144    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990146    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990148    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990151    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990153    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990156    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990159    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990161    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 15:12:19.997447 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990164    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990166    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990169    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990172    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990174    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990177    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990179    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990182    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990184    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990186    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990189    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990191    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990194    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990196    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990198    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990201    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990203    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990206    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990208    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990210    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990213    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 15:12:19.997897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990215    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990217    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990220    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990222    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990225    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990227    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990229    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990232    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990234    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990236    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990239    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990241    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990244    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990246    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990249    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990251    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990254    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990256    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990259    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 15:12:19.998376 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:19.990261    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 15:12:19.998847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.990266    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 15:12:19.998847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.990384    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 15:12:19.998847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.992419    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 15:12:19.998847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.993301    2579 server.go:1019] "Starting client certificate rotation"
Sep 19 15:12:19.998847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.993400    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 15:12:19.998847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.994241    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 15:12:19.998847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:19.998497    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 15:12:20.000253 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.000234    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 15:12:20.006039 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.006024    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 19 15:12:20.007581 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.007569    2579 log.go:25] "Validated CRI v1 image API"
Sep 19 15:12:20.008092 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.008081    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 15:12:20.010866 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.010850    2579 fs.go:135] Filesystem UUIDs: map[21a3a133-1b51-444c-9179-3b913ec682d2:/dev/nvme0n1p4 276fd87a-1f4d-406d-bd4a-be9fc97c9a36:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 19 15:12:20.010909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.010867    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 15:12:20.015285 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.015173    2579 manager.go:216] Machine: {Timestamp:2026-09-19 15:12:20.014723853 +0000 UTC m=+1.908287484 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200181 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec281835b6857d5274746766c9eb6419 SystemUUID:ec281835-b685-7d52-7474-6766c9eb6419 BootID:a262bda7-31ec-4efa-a11a-5ce60d487507 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:5a:bb:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:5a:bb:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:b3:28:5b:ff:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 15:12:20.015285 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.015276    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 15:12:20.015404 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.015371    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 15:12:20.017049 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.017007    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 15:12:20.017208 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.017054    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-0.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 15:12:20.018532 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.018519    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 15:12:20.018586 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.018537    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 15:12:20.018586 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.018547    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 15:12:20.018586 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.018562    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 15:12:20.018744 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.018658    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 15:12:20.022969 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.022955    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 15:12:20.023096 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.023085    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 15:12:20.023174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.023132    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 15:12:20.023174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.023151    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 15:12:20.023257 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.023183    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 15:12:20.023257 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.023196    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 19 15:12:20.023257 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.023223    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 15:12:20.023567 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.023544    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 15:12:20.023567 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.023567    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 15:12:20.025825 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.025687    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 15:12:20.027617 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.027573    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 15:12:20.030312 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.030298    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 15:12:20.030372 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.030327    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 15:12:20.031768 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.031750    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 15:12:20.031821 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.031781    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 15:12:20.031951 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.031940    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 15:12:20.031981 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.031956    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 15:12:20.031981 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.031962    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 15:12:20.031981 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.031968    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 15:12:20.031981 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.031976    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 15:12:20.031981 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.031981    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 15:12:20.032121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.031993    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 15:12:20.032121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.031998    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 15:12:20.032121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.032004    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 15:12:20.032121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.032009    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 15:12:20.032121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.032017    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 15:12:20.032121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.032021    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 15:12:20.032121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.032045    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 15:12:20.032121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.032050    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 15:12:20.035415 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.035402    2579 server.go:1268] "Started kubelet"
Sep 19 15:12:20.035486 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.035465    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 15:12:20.035551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.035503    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 15:12:20.035614 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.035573    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 15:12:20.036140 ip-10-0-141-0 systemd[1]: Started Kubernetes Kubelet.
Sep 19 15:12:20.036585 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.036503    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 15:12:20.038945 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.038927    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 15:12:20.042482 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.042463    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 15:12:20.042641 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.042628    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-0.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 15:12:20.043849 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.043834    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mr6jv"
Sep 19 15:12:20.045684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.045672    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 15:12:20.045684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.045682    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 15:12:20.045771 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.045731    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.046449 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046394    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 15:12:20.046449 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046446    2579 factory.go:55] Registering systemd factory
Sep 19 15:12:20.046583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046462    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 19 15:12:20.046583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046451    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 15:12:20.046583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046506    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 15:12:20.046583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046553    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 15:12:20.046583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046563    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 15:12:20.046791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046669    2579 factory.go:153] Registering CRI-O factory
Sep 19 15:12:20.046791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046679    2579 factory.go:223] Registration of the crio container factory successfully
Sep 19 15:12:20.046791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046730    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 15:12:20.046791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046748    2579 factory.go:103] Registering Raw factory
Sep 19 15:12:20.046791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.046761    2579 manager.go:1196] Started watching for new ooms in manager
Sep 19 15:12:20.047141 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.047129    2579 manager.go:349] Starting recovery of all containers
Sep 19 15:12:20.048705 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.048685    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mr6jv"
Sep 19 15:12:20.053899 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.053885    2579 manager.go:354] Recovery completed
Sep 19 15:12:20.067390 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.067371    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 15:12:20.071544 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.071530    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 15:12:20.071670 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.071655    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-0.ec2.internal\" not found" node="ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.074203 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.074186    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 15:12:20.074264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.074210    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 15:12:20.074264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.074219    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientPID"
Sep 19 15:12:20.074719 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.074709    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 15:12:20.074719 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.074718    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 15:12:20.074795 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.074721    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 15:12:20.074795 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.074733    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 15:12:20.075443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.075429    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 15:12:20.075506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.075460    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 15:12:20.075506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.075471    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientPID"
Sep 19 15:12:20.075880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.075868    2579 policy_none.go:49] "None policy: Start"
Sep 19 15:12:20.075921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.075884    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 15:12:20.075921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.075894    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 15:12:20.076872 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.076861    2579 policy_none.go:47] "Start"
Sep 19 15:12:20.113336 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.113178    2579 manager.go:337] "Starting Device Plugin manager"
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.113351    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.113364    2579 server.go:87] "Starting device plugin registration server"
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.113585    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.113618    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.113708    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.113784    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.113793    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.114308    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 15:12:20.129844 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.114358    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.163094 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.163066    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 15:12:20.164324 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.164306    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 15:12:20.164400 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.164359    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 15:12:20.164400 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.164382    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 15:12:20.164471 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.164423    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 15:12:20.166128 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.166112    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 15:12:20.214290 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.214262    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 15:12:20.215016 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.215002    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 15:12:20.215079 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.215026    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 15:12:20.215079 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.215036    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientPID"
Sep 19 15:12:20.215079 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.215056    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.223466 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.223419    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.223466 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.223439    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-0.ec2.internal\": node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.238188 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.238170    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.264626 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.264595    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal"]
Sep 19 15:12:20.264693 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.264682    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 15:12:20.265416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.265397    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 15:12:20.265492 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.265422    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 15:12:20.265492 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.265432    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientPID"
Sep 19 15:12:20.266677 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.266666    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 15:12:20.266796 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.266783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.266836 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.266811    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 15:12:20.267327 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.267314    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 15:12:20.267405 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.267339    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 15:12:20.267405 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.267318    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 15:12:20.267405 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.267352    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientPID"
Sep 19 15:12:20.267405 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.267365    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 15:12:20.267405 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.267380    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientPID"
Sep 19 15:12:20.268526 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.268508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.268596 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.268535    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 15:12:20.269218 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.269204    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 15:12:20.269295 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.269235    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 15:12:20.269295 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.269246    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeHasSufficientPID"
Sep 19 15:12:20.297028 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.297008    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-0.ec2.internal\" not found" node="ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.301196 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.301181    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-0.ec2.internal\" not found" node="ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.338885 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.338869    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.348165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.348143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90d1d9526d2ae5083b8d03f7ccdd81d9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-0.ec2.internal\" (UID: \"90d1d9526d2ae5083b8d03f7ccdd81d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.348239 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.348167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/889b87bbfc3b366c77fbd48e6a91bed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal\" (UID: \"889b87bbfc3b366c77fbd48e6a91bed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.348239 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.348193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889b87bbfc3b366c77fbd48e6a91bed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal\" (UID: \"889b87bbfc3b366c77fbd48e6a91bed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.439719 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.439692    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.449036 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.449017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90d1d9526d2ae5083b8d03f7ccdd81d9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-0.ec2.internal\" (UID: \"90d1d9526d2ae5083b8d03f7ccdd81d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.449080 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.449049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/889b87bbfc3b366c77fbd48e6a91bed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal\" (UID: \"889b87bbfc3b366c77fbd48e6a91bed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.449080 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.449072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889b87bbfc3b366c77fbd48e6a91bed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal\" (UID: \"889b87bbfc3b366c77fbd48e6a91bed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.449139 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.449085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90d1d9526d2ae5083b8d03f7ccdd81d9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-0.ec2.internal\" (UID: \"90d1d9526d2ae5083b8d03f7ccdd81d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.449139 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.449114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/889b87bbfc3b366c77fbd48e6a91bed6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal\" (UID: \"889b87bbfc3b366c77fbd48e6a91bed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.449139 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.449118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889b87bbfc3b366c77fbd48e6a91bed6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal\" (UID: \"889b87bbfc3b366c77fbd48e6a91bed6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.540493 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.540444    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.601921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.601903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.606474 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.606364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:20.641062 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.641040    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.741609 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.741582    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.842094 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.842034    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.942675 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:20.942644    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:20.993075 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.993056    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 15:12:20.993427 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.993187    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 15:12:20.993427 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:20.993216    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 15:12:21.043520 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:21.043497    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:21.054911 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.054884    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 15:07:20 +0000 UTC" deadline="2028-05-04 19:42:29.646440626 +0000 UTC"
Sep 19 15:12:21.054911 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.054909    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14236h30m8.591534654s"
Sep 19 15:12:21.143849 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:21.143793    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:21.223314 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:21.223278    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889b87bbfc3b366c77fbd48e6a91bed6.slice/crio-49b2a24505590a4eadf9fb3ddba32835a9349d196376db25381392a6cb591175 WatchSource:0}: Error finding container 49b2a24505590a4eadf9fb3ddba32835a9349d196376db25381392a6cb591175: Status 404 returned error can't find the container with id 49b2a24505590a4eadf9fb3ddba32835a9349d196376db25381392a6cb591175
Sep 19 15:12:21.223849 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:21.223832    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d1d9526d2ae5083b8d03f7ccdd81d9.slice/crio-4932544fe3bd8ddf7968e27658eea39b9897403f4e507a6e93a67bd7fb0e62c2 WatchSource:0}: Error finding container 4932544fe3bd8ddf7968e27658eea39b9897403f4e507a6e93a67bd7fb0e62c2: Status 404 returned error can't find the container with id 4932544fe3bd8ddf7968e27658eea39b9897403f4e507a6e93a67bd7fb0e62c2
Sep 19 15:12:21.228914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.228898    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 15:12:21.244238 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:21.244221    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:21.344765 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:21.344739    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-0.ec2.internal\" not found"
Sep 19 15:12:21.422134 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.422086    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 15:12:21.445858 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.445838    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:21.456860 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.456839    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 15:12:21.458519 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.458505    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal"
Sep 19 15:12:21.466852 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.466838    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 15:12:21.502659 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:21.502641    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 15:12:22.025247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.025220    2579 apiserver.go:52] "Watching apiserver"
Sep 19 15:12:22.029842 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.029820    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 15:12:22.032198 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.032167    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-7l4fc","openshift-cluster-node-tuning-operator/tuned-kk64t","openshift-multus/multus-additional-cni-plugins-8t2m7","openshift-multus/multus-dssqn","openshift-multus/network-metrics-daemon-7vfsx","openshift-network-operator/iptables-alerter-zwl8d","openshift-ovn-kubernetes/ovnkube-node-j5v77","kube-system/global-pull-secret-syncer-zmfsl","kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth","openshift-dns/node-resolver-jz2tb","openshift-image-registry/node-ca-d28d4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal","openshift-network-diagnostics/network-check-target-4mr5z"]
Sep 19 15:12:22.034218 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.034196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.035582 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.035558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.036154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.036131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 15:12:22.036443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.036427    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vvp9d\""
Sep 19 15:12:22.036843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.036803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.037237 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.037207    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 15:12:22.037308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.037291    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 15:12:22.037344 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.037325    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 15:12:22.037514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.037495    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 15:12:22.037514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.037509    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 15:12:22.037704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.037565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 15:12:22.038003 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.037788    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 15:12:22.038003 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.037847    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-92tjb\""
Sep 19 15:12:22.038522 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.038328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.038522 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.038379    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 15:12:22.038685 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.038525    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 15:12:22.038685 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.038647    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 15:12:22.038779 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.038734    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 15:12:22.038779 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.038756    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 15:12:22.039001 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.038985    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 15:12:22.039134 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.039028    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nz4tp\""
Sep 19 15:12:22.039639 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.039623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:22.039738 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.039683    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:22.039860 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.039812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 15:12:22.040142 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.040035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4zs8l\""
Sep 19 15:12:22.041065 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.041032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.042596 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.042562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:22.042704 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.042625    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:22.043027 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.043009    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lm6wq\""
Sep 19 15:12:22.043114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.043035    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 15:12:22.043114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.043046    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 15:12:22.043313 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.043298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 15:12:22.044081 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.044027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.044163 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.044134    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:22.045827 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.045470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.050165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.046857    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 15:12:22.050165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.047557    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 15:12:22.050165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.047734    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 15:12:22.050165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.047892    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9jc4h\""
Sep 19 15:12:22.050165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.048768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.050165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.049510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:22.050165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.049831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.050505 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.050487    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 15:12:22.050683 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.050664    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vtsv4\""
Sep 19 15:12:22.050757 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.050736    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 15:12:22.050902 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.050882    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 15:12:22.051482 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.051467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 15:12:22.051756 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.051649    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 15:12:22.051756 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.051650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x27p5\""
Sep 19 15:12:22.051891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.051760    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 15:12:22.051943 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.051904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 15:12:22.051943 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.051925    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 15:12:22.052034 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.052000    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nl87t\""
Sep 19 15:12:22.058235 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.058323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysctl-conf\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.058323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-sys\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.058323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-lib-modules\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.058467 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.058467 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-k8s-cni-cncf-io\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.058467 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovn-node-metrics-cert\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.058590 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-tmp-dir\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.058590 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-var-lib-kubelet\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.058590 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-node-log\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.058590 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9e06119-2d4a-42ce-b018-9f947d18ad98-dbus\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-socket-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-system-cni-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-cni-bin\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysconfig\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61a4cb3c-f020-4393-9222-a814cece3949-agent-certs\") pod \"konnectivity-agent-7l4fc\" (UID: \"61a4cb3c-f020-4393-9222-a814cece3949\") " pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-device-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-hostroot\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.058843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-modprobe-d\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-var-lib-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-cni-netd\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcms5\" (UniqueName: \"kubernetes.io/projected/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-kube-api-access-fcms5\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-kubernetes\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-tuned\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ca6df2a-1979-42bb-80ea-4930186624e9-iptables-alerter-script\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.058997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ca6df2a-1979-42bb-80ea-4930186624e9-host-slash\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgqx\" (UniqueName: \"kubernetes.io/projected/5ca6df2a-1979-42bb-80ea-4930186624e9-kube-api-access-hcgqx\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-systemd\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-host\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-log-socket\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vq8\" (UniqueName: \"kubernetes.io/projected/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-kube-api-access-x4vq8\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.059315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nsht\" (UniqueName: \"kubernetes.io/projected/da06acaf-ee67-45cd-a1bb-be5e77cf764d-kube-api-access-6nsht\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-run\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61a4cb3c-f020-4393-9222-a814cece3949-konnectivity-ca\") pod \"konnectivity-agent-7l4fc\" (UID: \"61a4cb3c-f020-4393-9222-a814cece3949\") " pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-hosts-file\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-system-cni-dir\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-daemon-config\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-multus-certs\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-etc-kubernetes\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-kubelet\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-slash\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-run-netns\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovnkube-script-lib\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9e06119-2d4a-42ce-b018-9f947d18ad98-kubelet-config\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-registration-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.060030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.059849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-sys-fs\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.060726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.060144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cnibin\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.060726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.060196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc9j7\" (UniqueName: \"kubernetes.io/projected/e261dff5-21eb-4a06-9996-349666d1fd9f-kube-api-access-jc9j7\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:22.060726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.060238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysctl-d\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.060726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.060270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f25e-aa96-49eb-b284-4037a2add73b-tmp\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.060726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.060308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-systemd-units\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.060726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.060337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-ovn\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.060726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.060362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.060726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.060500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n866s\" (UniqueName: \"kubernetes.io/projected/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-kube-api-access-n866s\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.061069 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-socket-dir-parent\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.061069 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:22.061166 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.061166 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-env-overrides\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.061166 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-cni-bin\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.061385 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-cni-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.061385 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-cnibin\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.061484 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-kubelet\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.061484 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-etc-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.061484 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovnkube-config\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-cni-binary-copy\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-netns\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-cni-multus\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-conf-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxw9\" (UniqueName: \"kubernetes.io/projected/fbc7f25e-aa96-49eb-b284-4037a2add73b-kube-api-access-rdxw9\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.061872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvblg\" (UniqueName: \"kubernetes.io/projected/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-kube-api-access-qvblg\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.062163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-systemd\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.062838    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 15:12:22.063154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.063060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:22.072283 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.072267    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgb9b"
Sep 19 15:12:22.079551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.079530    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pgb9b"
Sep 19 15:12:22.147426 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.147404    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 15:12:22.163826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.163765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-systemd\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.163826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.163793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:22.163981 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.163808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.163981 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.163857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysctl-conf\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.163981 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.163897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-systemd\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.163981 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.163961    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:22.164160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysctl-conf\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.164160 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.164030    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret podName:d9e06119-2d4a-42ce-b018-9f947d18ad98 nodeName:}" failed. No retries permitted until 2026-09-19 15:12:22.663999733 +0000 UTC m=+4.557563360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret") pod "global-pull-secret-syncer-zmfsl" (UID: "d9e06119-2d4a-42ce-b018-9f947d18ad98") : object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:22.164160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-sys\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.164160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-lib-modules\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-k8s-cni-cncf-io\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-sys\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovn-node-metrics-cert\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-tmp-dir\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-var-lib-kubelet\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-k8s-cni-cncf-io\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-node-log\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-lib-modules\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-node-log\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9e06119-2d4a-42ce-b018-9f947d18ad98-dbus\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-var-lib-kubelet\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-socket-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.164416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-system-cni-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-cni-bin\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysconfig\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61a4cb3c-f020-4393-9222-a814cece3949-agent-certs\") pod \"konnectivity-agent-7l4fc\" (UID: \"61a4cb3c-f020-4393-9222-a814cece3949\") " pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-cni-bin\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysconfig\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-device-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-socket-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-hostroot\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-tmp-dir\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164582    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-modprobe-d\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-system-cni-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-var-lib-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-hostroot\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-cni-netd\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.165098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcms5\" (UniqueName: \"kubernetes.io/projected/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-kube-api-access-fcms5\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-var-lib-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-kubernetes\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-tuned\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-modprobe-d\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ca6df2a-1979-42bb-80ea-4930186624e9-iptables-alerter-script\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ca6df2a-1979-42bb-80ea-4930186624e9-host-slash\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-device-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgqx\" (UniqueName: \"kubernetes.io/projected/5ca6df2a-1979-42bb-80ea-4930186624e9-kube-api-access-hcgqx\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-cni-netd\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-systemd\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-host\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.165931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-log-socket\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-log-socket\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vq8\" (UniqueName: \"kubernetes.io/projected/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-kube-api-access-x4vq8\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nsht\" (UniqueName: \"kubernetes.io/projected/da06acaf-ee67-45cd-a1bb-be5e77cf764d-kube-api-access-6nsht\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-run\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61a4cb3c-f020-4393-9222-a814cece3949-konnectivity-ca\") pod \"konnectivity-agent-7l4fc\" (UID: \"61a4cb3c-f020-4393-9222-a814cece3949\") " pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-hosts-file\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-host\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.164937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-kubernetes\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-system-cni-dir\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-daemon-config\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-multus-certs\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-etc-kubernetes\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8thr\" (UniqueName: \"kubernetes.io/projected/500406fa-6c94-45a7-a391-5c758bdd727b-kube-api-access-n8thr\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.166670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-kubelet\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-slash\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-run-netns\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovnkube-script-lib\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9e06119-2d4a-42ce-b018-9f947d18ad98-kubelet-config\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-registration-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-sys-fs\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cnibin\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc9j7\" (UniqueName: \"kubernetes.io/projected/e261dff5-21eb-4a06-9996-349666d1fd9f-kube-api-access-jc9j7\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysctl-d\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f25e-aa96-49eb-b284-4037a2add73b-tmp\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-systemd-units\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-ovn\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n866s\" (UniqueName: \"kubernetes.io/projected/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-kube-api-access-n866s\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-socket-dir-parent\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.167418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-env-overrides\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-socket-dir-parent\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/500406fa-6c94-45a7-a391-5c758bdd727b-serviceca\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-daemon-config\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-cni-bin\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-etc-kubernetes\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-cni-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-cni-bin\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ca6df2a-1979-42bb-80ea-4930186624e9-iptables-alerter-script\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.165150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-systemd\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-multus-certs\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-kubelet\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-cnibin\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9e06119-2d4a-42ce-b018-9f947d18ad98-kubelet-config\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-registration-dir\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.173221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da06acaf-ee67-45cd-a1bb-be5e77cf764d-sys-fs\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-cnibin\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-cnibin\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ca6df2a-1979-42bb-80ea-4930186624e9-host-slash\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-kubelet\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-env-overrides\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-slash\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-cni-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-run-ovn\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-systemd-units\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-host-run-netns\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovnkube-script-lib\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-kubelet\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-system-cni-dir\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.166664    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.166899    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:12:22.666880425 +0000 UTC m=+4.560444056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:22.174066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-run\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.166963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-hosts-file\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167038    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61a4cb3c-f020-4393-9222-a814cece3949-konnectivity-ca\") pod \"konnectivity-agent-7l4fc\" (UID: \"61a4cb3c-f020-4393-9222-a814cece3949\") " pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/500406fa-6c94-45a7-a391-5c758bdd727b-host\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-etc-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-sysctl-d\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-etc-openvswitch\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovnkube-config\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9e06119-2d4a-42ce-b018-9f947d18ad98-dbus\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-cni-binary-copy\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-netns\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-cni-multus\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-conf-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxw9\" (UniqueName: \"kubernetes.io/projected/fbc7f25e-aa96-49eb-b284-4037a2add73b-kube-api-access-rdxw9\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvblg\" (UniqueName: \"kubernetes.io/projected/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-kube-api-access-qvblg\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-cni-binary-copy\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovnkube-config\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.174834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-run-netns\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-multus-conf-dir\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.167819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-host-var-lib-cni-multus\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.168583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-ovn-node-metrics-cert\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.168912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61a4cb3c-f020-4393-9222-a814cece3949-agent-certs\") pod \"konnectivity-agent-7l4fc\" (UID: \"61a4cb3c-f020-4393-9222-a814cece3949\") " pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.169460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f25e-aa96-49eb-b284-4037a2add73b-etc-tuned\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.170217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f25e-aa96-49eb-b284-4037a2add73b-tmp\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.170339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal" event={"ID":"90d1d9526d2ae5083b8d03f7ccdd81d9","Type":"ContainerStarted","Data":"4932544fe3bd8ddf7968e27658eea39b9897403f4e507a6e93a67bd7fb0e62c2"}
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.171588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal" event={"ID":"889b87bbfc3b366c77fbd48e6a91bed6","Type":"ContainerStarted","Data":"49b2a24505590a4eadf9fb3ddba32835a9349d196376db25381392a6cb591175"}
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.172214    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.172229    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.172238    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8dscp for pod openshift-network-diagnostics/network-check-target-4mr5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:22.175497 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.172294    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp podName:ff785d9b-cbc1-48e2-9af1-53f35353cd9a nodeName:}" failed. No retries permitted until 2026-09-19 15:12:22.672281519 +0000 UTC m=+4.565845138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8dscp" (UniqueName: "kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp") pod "network-check-target-4mr5z" (UID: "ff785d9b-cbc1-48e2-9af1-53f35353cd9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:22.177049 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.177022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vq8\" (UniqueName: \"kubernetes.io/projected/b9ecfb66-d607-4989-8f1e-3651e26cc2a3-kube-api-access-x4vq8\") pod \"node-resolver-jz2tb\" (UID: \"b9ecfb66-d607-4989-8f1e-3651e26cc2a3\") " pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.177182 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.177098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n866s\" (UniqueName: \"kubernetes.io/projected/5ab9acdc-6a7c-4407-bfdf-85b2f096455f-kube-api-access-n866s\") pod \"multus-additional-cni-plugins-8t2m7\" (UID: \"5ab9acdc-6a7c-4407-bfdf-85b2f096455f\") " pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.177182 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.177098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxw9\" (UniqueName: \"kubernetes.io/projected/fbc7f25e-aa96-49eb-b284-4037a2add73b-kube-api-access-rdxw9\") pod \"tuned-kk64t\" (UID: \"fbc7f25e-aa96-49eb-b284-4037a2add73b\") " pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.177182 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.177121    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nsht\" (UniqueName: \"kubernetes.io/projected/da06acaf-ee67-45cd-a1bb-be5e77cf764d-kube-api-access-6nsht\") pod \"aws-ebs-csi-driver-node-sfsth\" (UID: \"da06acaf-ee67-45cd-a1bb-be5e77cf764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.178309 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.178284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgqx\" (UniqueName: \"kubernetes.io/projected/5ca6df2a-1979-42bb-80ea-4930186624e9-kube-api-access-hcgqx\") pod \"iptables-alerter-zwl8d\" (UID: \"5ca6df2a-1979-42bb-80ea-4930186624e9\") " pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.179848 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.179826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc9j7\" (UniqueName: \"kubernetes.io/projected/e261dff5-21eb-4a06-9996-349666d1fd9f-kube-api-access-jc9j7\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:22.180026 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.180005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcms5\" (UniqueName: \"kubernetes.io/projected/d9d4c00c-9b8d-485f-836c-6af14fe6cd6e-kube-api-access-fcms5\") pod \"ovnkube-node-j5v77\" (UID: \"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.183101 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.183083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvblg\" (UniqueName: \"kubernetes.io/projected/658ee5aa-e3bf-4397-b6ab-86bdae6793a5-kube-api-access-qvblg\") pod \"multus-dssqn\" (UID: \"658ee5aa-e3bf-4397-b6ab-86bdae6793a5\") " pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.267595 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.267561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/500406fa-6c94-45a7-a391-5c758bdd727b-serviceca\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.267595 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.267597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/500406fa-6c94-45a7-a391-5c758bdd727b-host\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.267826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.267663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/500406fa-6c94-45a7-a391-5c758bdd727b-host\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.267826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.267733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8thr\" (UniqueName: \"kubernetes.io/projected/500406fa-6c94-45a7-a391-5c758bdd727b-kube-api-access-n8thr\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.268039 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.268016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/500406fa-6c94-45a7-a391-5c758bdd727b-serviceca\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.276060 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.276035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8thr\" (UniqueName: \"kubernetes.io/projected/500406fa-6c94-45a7-a391-5c758bdd727b-kube-api-access-n8thr\") pod \"node-ca-d28d4\" (UID: \"500406fa-6c94-45a7-a391-5c758bdd727b\") " pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.346059 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.346024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:22.363534 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.363515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jz2tb"
Sep 19 15:12:22.372113 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.372092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8t2m7"
Sep 19 15:12:22.377674 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.377656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dssqn"
Sep 19 15:12:22.385031 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.385013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zwl8d"
Sep 19 15:12:22.389368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.389351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kk64t"
Sep 19 15:12:22.396045 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.396028    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth"
Sep 19 15:12:22.401497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.401481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:22.406887 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.406870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d28d4"
Sep 19 15:12:22.524003 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.523969    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 15:12:22.670065 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.670035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:22.670223 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.670095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:22.670223 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.670159    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:22.670223 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.670182    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:22.670223 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.670219    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret podName:d9e06119-2d4a-42ce-b018-9f947d18ad98 nodeName:}" failed. No retries permitted until 2026-09-19 15:12:23.670202804 +0000 UTC m=+5.563766423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret") pod "global-pull-secret-syncer-zmfsl" (UID: "d9e06119-2d4a-42ce-b018-9f947d18ad98") : object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:22.670408 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.670235    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:12:23.670227048 +0000 UTC m=+5.563790666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:22.771376 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:22.771343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:22.771542 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.771515    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 15:12:22.771542 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.771535    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 15:12:22.771631 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.771544    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8dscp for pod openshift-network-diagnostics/network-check-target-4mr5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:22.771631 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:22.771596    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp podName:ff785d9b-cbc1-48e2-9af1-53f35353cd9a nodeName:}" failed. No retries permitted until 2026-09-19 15:12:23.771578995 +0000 UTC m=+5.665142619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8dscp" (UniqueName: "kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp") pod "network-check-target-4mr5z" (UID: "ff785d9b-cbc1-48e2-9af1-53f35353cd9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:23.009327 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.009299    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc7f25e_aa96_49eb_b284_4037a2add73b.slice/crio-4ad8da31ddee54696574eed9086308ff9036dfc5b7e304125cc99543d26c135f WatchSource:0}: Error finding container 4ad8da31ddee54696574eed9086308ff9036dfc5b7e304125cc99543d26c135f: Status 404 returned error can't find the container with id 4ad8da31ddee54696574eed9086308ff9036dfc5b7e304125cc99543d26c135f
Sep 19 15:12:23.010550 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.010493    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a4cb3c_f020_4393_9222_a814cece3949.slice/crio-6465f79d3e1530df7f23dce52f4111e60eb19f0e5f617ec0236ada4430b13874 WatchSource:0}: Error finding container 6465f79d3e1530df7f23dce52f4111e60eb19f0e5f617ec0236ada4430b13874: Status 404 returned error can't find the container with id 6465f79d3e1530df7f23dce52f4111e60eb19f0e5f617ec0236ada4430b13874
Sep 19 15:12:23.010960 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.010941    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d4c00c_9b8d_485f_836c_6af14fe6cd6e.slice/crio-39c6ecc3323a0b12956a3ce2315304d29320a61f45cf91de808324a862a2a668 WatchSource:0}: Error finding container 39c6ecc3323a0b12956a3ce2315304d29320a61f45cf91de808324a862a2a668: Status 404 returned error can't find the container with id 39c6ecc3323a0b12956a3ce2315304d29320a61f45cf91de808324a862a2a668
Sep 19 15:12:23.011623 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.011587    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca6df2a_1979_42bb_80ea_4930186624e9.slice/crio-27bcba6fbdd2e9aae171cfbd874ea18f6793a4b4a729918ad05bc33ab82be6bd WatchSource:0}: Error finding container 27bcba6fbdd2e9aae171cfbd874ea18f6793a4b4a729918ad05bc33ab82be6bd: Status 404 returned error can't find the container with id 27bcba6fbdd2e9aae171cfbd874ea18f6793a4b4a729918ad05bc33ab82be6bd
Sep 19 15:12:23.013837 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.013272    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda06acaf_ee67_45cd_a1bb_be5e77cf764d.slice/crio-9e63391f20639b1666075165bf3f36ac3227fe166df65fa2ce17a016344cdb80 WatchSource:0}: Error finding container 9e63391f20639b1666075165bf3f36ac3227fe166df65fa2ce17a016344cdb80: Status 404 returned error can't find the container with id 9e63391f20639b1666075165bf3f36ac3227fe166df65fa2ce17a016344cdb80
Sep 19 15:12:23.017129 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.017053    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658ee5aa_e3bf_4397_b6ab_86bdae6793a5.slice/crio-7847db7004af03a056e68126b9ab7959fb58d7f531a3db4d8bbabfc4e7f799a4 WatchSource:0}: Error finding container 7847db7004af03a056e68126b9ab7959fb58d7f531a3db4d8bbabfc4e7f799a4: Status 404 returned error can't find the container with id 7847db7004af03a056e68126b9ab7959fb58d7f531a3db4d8bbabfc4e7f799a4
Sep 19 15:12:23.017496 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.017476    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ecfb66_d607_4989_8f1e_3651e26cc2a3.slice/crio-3bac91faaa9c54398aab43caa3ae91722828a2502d394bca715c14fede2f9d4a WatchSource:0}: Error finding container 3bac91faaa9c54398aab43caa3ae91722828a2502d394bca715c14fede2f9d4a: Status 404 returned error can't find the container with id 3bac91faaa9c54398aab43caa3ae91722828a2502d394bca715c14fede2f9d4a
Sep 19 15:12:23.020412 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.020392    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500406fa_6c94_45a7_a391_5c758bdd727b.slice/crio-e9066af2ea55b18fa16e9a7955a53bf59650b9a6aca105b30d4931c3ef8fbc01 WatchSource:0}: Error finding container e9066af2ea55b18fa16e9a7955a53bf59650b9a6aca105b30d4931c3ef8fbc01: Status 404 returned error can't find the container with id e9066af2ea55b18fa16e9a7955a53bf59650b9a6aca105b30d4931c3ef8fbc01
Sep 19 15:12:23.020681 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:23.020564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab9acdc_6a7c_4407_bfdf_85b2f096455f.slice/crio-f18707f8b91e5206ed53ae76a8e855d1af2979a93927a240e7c4af2ede64f378 WatchSource:0}: Error finding container f18707f8b91e5206ed53ae76a8e855d1af2979a93927a240e7c4af2ede64f378: Status 404 returned error can't find the container with id f18707f8b91e5206ed53ae76a8e855d1af2979a93927a240e7c4af2ede64f378
Sep 19 15:12:23.081501 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.081474    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 15:07:22 +0000 UTC" deadline="2028-07-03 03:47:00.703184332 +0000 UTC"
Sep 19 15:12:23.081501 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.081500    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15660h34m37.621687646s"
Sep 19 15:12:23.165248 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.165221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:23.165371 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.165327    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:23.174573 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.174546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7l4fc" event={"ID":"61a4cb3c-f020-4393-9222-a814cece3949","Type":"ContainerStarted","Data":"6465f79d3e1530df7f23dce52f4111e60eb19f0e5f617ec0236ada4430b13874"}
Sep 19 15:12:23.175726 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.175703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dssqn" event={"ID":"658ee5aa-e3bf-4397-b6ab-86bdae6793a5","Type":"ContainerStarted","Data":"7847db7004af03a056e68126b9ab7959fb58d7f531a3db4d8bbabfc4e7f799a4"}
Sep 19 15:12:23.176794 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.176775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth" event={"ID":"da06acaf-ee67-45cd-a1bb-be5e77cf764d","Type":"ContainerStarted","Data":"9e63391f20639b1666075165bf3f36ac3227fe166df65fa2ce17a016344cdb80"}
Sep 19 15:12:23.177846 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.177825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zwl8d" event={"ID":"5ca6df2a-1979-42bb-80ea-4930186624e9","Type":"ContainerStarted","Data":"27bcba6fbdd2e9aae171cfbd874ea18f6793a4b4a729918ad05bc33ab82be6bd"}
Sep 19 15:12:23.178898 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.178872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"39c6ecc3323a0b12956a3ce2315304d29320a61f45cf91de808324a862a2a668"}
Sep 19 15:12:23.180101 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.180079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kk64t" event={"ID":"fbc7f25e-aa96-49eb-b284-4037a2add73b","Type":"ContainerStarted","Data":"4ad8da31ddee54696574eed9086308ff9036dfc5b7e304125cc99543d26c135f"}
Sep 19 15:12:23.182350 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.182130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal" event={"ID":"90d1d9526d2ae5083b8d03f7ccdd81d9","Type":"ContainerStarted","Data":"50939cf8cbb2239c62059d5e3727680c67ec1d99b7932160c6bc3b9214e8a5c4"}
Sep 19 15:12:23.183266 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.183247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d28d4" event={"ID":"500406fa-6c94-45a7-a391-5c758bdd727b","Type":"ContainerStarted","Data":"e9066af2ea55b18fa16e9a7955a53bf59650b9a6aca105b30d4931c3ef8fbc01"}
Sep 19 15:12:23.184399 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.184362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" event={"ID":"5ab9acdc-6a7c-4407-bfdf-85b2f096455f","Type":"ContainerStarted","Data":"f18707f8b91e5206ed53ae76a8e855d1af2979a93927a240e7c4af2ede64f378"}
Sep 19 15:12:23.185408 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.185389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jz2tb" event={"ID":"b9ecfb66-d607-4989-8f1e-3651e26cc2a3","Type":"ContainerStarted","Data":"3bac91faaa9c54398aab43caa3ae91722828a2502d394bca715c14fede2f9d4a"}
Sep 19 15:12:23.678708 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.678080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:23.678708 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.678152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:23.678708 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.678267    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:23.678708 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.678325    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:12:25.678306386 +0000 UTC m=+7.571870010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:23.678708 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.678711    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:23.679066 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.678761    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret podName:d9e06119-2d4a-42ce-b018-9f947d18ad98 nodeName:}" failed. No retries permitted until 2026-09-19 15:12:25.678745689 +0000 UTC m=+7.572309314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret") pod "global-pull-secret-syncer-zmfsl" (UID: "d9e06119-2d4a-42ce-b018-9f947d18ad98") : object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:23.779182 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:23.779130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:23.779354 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.779307    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 15:12:23.779354 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.779327    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 15:12:23.779354 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.779339    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8dscp for pod openshift-network-diagnostics/network-check-target-4mr5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:23.779523 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:23.779398    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp podName:ff785d9b-cbc1-48e2-9af1-53f35353cd9a nodeName:}" failed. No retries permitted until 2026-09-19 15:12:25.779380371 +0000 UTC m=+7.672943994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8dscp" (UniqueName: "kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp") pod "network-check-target-4mr5z" (UID: "ff785d9b-cbc1-48e2-9af1-53f35353cd9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:24.081920 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:24.081795    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 15:07:22 +0000 UTC" deadline="2028-04-30 04:18:20.195364875 +0000 UTC"
Sep 19 15:12:24.081920 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:24.081851    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14125h5m56.113517804s"
Sep 19 15:12:24.169713 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:24.169015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:24.169713 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:24.169146    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:24.169713 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:24.169508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:24.169713 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:24.169594    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:24.198812 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:24.198782    2579 generic.go:356] "Generic (PLEG): container finished" podID="889b87bbfc3b366c77fbd48e6a91bed6" containerID="12f385822d8309f2f37a8ac122b3e037f2878c0fd94d02d03f451e4c83b131b0" exitCode=0
Sep 19 15:12:24.199704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:24.199673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal" event={"ID":"889b87bbfc3b366c77fbd48e6a91bed6","Type":"ContainerDied","Data":"12f385822d8309f2f37a8ac122b3e037f2878c0fd94d02d03f451e4c83b131b0"}
Sep 19 15:12:24.215263 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:24.215210    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-0.ec2.internal" podStartSLOduration=3.215191433 podStartE2EDuration="3.215191433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:23.194154972 +0000 UTC m=+5.087718613" watchObservedRunningTime="2026-09-19 15:12:24.215191433 +0000 UTC m=+6.108755074"
Sep 19 15:12:25.165196 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:25.165157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:25.165847 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.165283    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:25.210159 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:25.210107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal" event={"ID":"889b87bbfc3b366c77fbd48e6a91bed6","Type":"ContainerStarted","Data":"dae238899b8b5c82c4350234d0548bdec6aefa24417ce106b3b5c6f160c385e5"}
Sep 19 15:12:25.694676 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:25.694633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:25.694843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:25.694699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:25.694928 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.694848    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:25.694928 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.694911    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret podName:d9e06119-2d4a-42ce-b018-9f947d18ad98 nodeName:}" failed. No retries permitted until 2026-09-19 15:12:29.694892697 +0000 UTC m=+11.588456331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret") pod "global-pull-secret-syncer-zmfsl" (UID: "d9e06119-2d4a-42ce-b018-9f947d18ad98") : object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:25.695271 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.695146    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:25.695271 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.695210    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:12:29.695191659 +0000 UTC m=+11.588755294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:25.796576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:25.795960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:25.796576 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.796158    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 15:12:25.796576 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.796181    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 15:12:25.796576 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.796193    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8dscp for pod openshift-network-diagnostics/network-check-target-4mr5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:25.796576 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:25.796251    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp podName:ff785d9b-cbc1-48e2-9af1-53f35353cd9a nodeName:}" failed. No retries permitted until 2026-09-19 15:12:29.796232128 +0000 UTC m=+11.689795775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8dscp" (UniqueName: "kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp") pod "network-check-target-4mr5z" (UID: "ff785d9b-cbc1-48e2-9af1-53f35353cd9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:26.168130 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:26.167594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:26.168130 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:26.167759    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:26.168130 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:26.167820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:26.168130 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:26.167958    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:27.168792 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:27.168200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:27.168792 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:27.168344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:28.165377 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:28.165344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:28.165377 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:28.165369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:28.165625 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:28.165487    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:28.165625 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:28.165590    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:29.165374 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:29.165337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:29.165822 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.165482    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:29.734677 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:29.734645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:29.734846 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:29.734708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:29.734846 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.734795    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:29.734846 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.734795    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:29.734958 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.734850    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret podName:d9e06119-2d4a-42ce-b018-9f947d18ad98 nodeName:}" failed. No retries permitted until 2026-09-19 15:12:37.734832125 +0000 UTC m=+19.628395757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret") pod "global-pull-secret-syncer-zmfsl" (UID: "d9e06119-2d4a-42ce-b018-9f947d18ad98") : object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:29.734958 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.734865    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:12:37.734857109 +0000 UTC m=+19.628420728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:29.835576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:29.835535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:29.835730 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.835695    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 15:12:29.835730 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.835712    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 15:12:29.835730 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.835721    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8dscp for pod openshift-network-diagnostics/network-check-target-4mr5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:29.835874 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:29.835762    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp podName:ff785d9b-cbc1-48e2-9af1-53f35353cd9a nodeName:}" failed. No retries permitted until 2026-09-19 15:12:37.835749783 +0000 UTC m=+19.729313401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8dscp" (UniqueName: "kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp") pod "network-check-target-4mr5z" (UID: "ff785d9b-cbc1-48e2-9af1-53f35353cd9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:30.166687 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:30.166615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:30.167053 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:30.166733    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:30.167163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:30.167080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:30.167215 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:30.167161    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:31.165633 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:31.165554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:31.165793 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:31.165663    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:32.165568 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:32.165529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:32.166081 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:32.165529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:32.166081 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:32.165679    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:32.166081 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:32.165713    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:33.165131 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:33.165049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:33.165276 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:33.165176    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:34.165171 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:34.165126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:34.165171 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:34.165170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:34.165708 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:34.165273    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:34.165708 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:34.165426    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:35.164841 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:35.164807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:35.165019 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:35.164929    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:36.165000 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:36.164965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:36.165408 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:36.164965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:36.165408 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:36.165129    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:36.165408 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:36.165168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:37.164614 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:37.164570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:37.164799 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.164738    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:37.795584 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:37.795547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:37.796010 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:37.795638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:37.796010 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.795742    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:37.796010 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.795810    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret podName:d9e06119-2d4a-42ce-b018-9f947d18ad98 nodeName:}" failed. No retries permitted until 2026-09-19 15:12:53.795794589 +0000 UTC m=+35.689358207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret") pod "global-pull-secret-syncer-zmfsl" (UID: "d9e06119-2d4a-42ce-b018-9f947d18ad98") : object "kube-system"/"original-pull-secret" not registered
Sep 19 15:12:37.796010 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.795742    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:37.796010 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.795881    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:12:53.795867443 +0000 UTC m=+35.689431077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 15:12:37.896582 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:37.896531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:37.896765 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.896714    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 15:12:37.896765 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.896733    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 15:12:37.896765 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.896746    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8dscp for pod openshift-network-diagnostics/network-check-target-4mr5z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:37.896899 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:37.896807    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp podName:ff785d9b-cbc1-48e2-9af1-53f35353cd9a nodeName:}" failed. No retries permitted until 2026-09-19 15:12:53.896787634 +0000 UTC m=+35.790351252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8dscp" (UniqueName: "kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp") pod "network-check-target-4mr5z" (UID: "ff785d9b-cbc1-48e2-9af1-53f35353cd9a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 15:12:38.164686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:38.164592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:38.164845 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:38.164759    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:38.164891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:38.164855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:38.164982 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:38.164957    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:39.164865 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:39.164831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:39.165284 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:39.164943    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:40.165581 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:40.165552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:40.165920 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:40.165646    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:40.165920 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:40.165747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:40.165920 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:40.165860    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:41.165311 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.165041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:41.165450 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:41.165345    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:41.241895 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.241863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d28d4" event={"ID":"500406fa-6c94-45a7-a391-5c758bdd727b","Type":"ContainerStarted","Data":"1c09650d3a534cedee41bd0d412a43adc689cb1500fe53a9a8a32e070001eac1"}
Sep 19 15:12:41.243372 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.243350    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ab9acdc-6a7c-4407-bfdf-85b2f096455f" containerID="5e4509d8bdf0785c625306c5801a64656d546cfae6ec4140c94e8ebd899427ba" exitCode=0
Sep 19 15:12:41.243459 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.243400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" event={"ID":"5ab9acdc-6a7c-4407-bfdf-85b2f096455f","Type":"ContainerDied","Data":"5e4509d8bdf0785c625306c5801a64656d546cfae6ec4140c94e8ebd899427ba"}
Sep 19 15:12:41.244826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.244757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jz2tb" event={"ID":"b9ecfb66-d607-4989-8f1e-3651e26cc2a3","Type":"ContainerStarted","Data":"0feec35754f06f1ae7483fe4c07e4047b2f1de292aa9b309727747ec5ab3cdee"}
Sep 19 15:12:41.246172 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.246150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7l4fc" event={"ID":"61a4cb3c-f020-4393-9222-a814cece3949","Type":"ContainerStarted","Data":"c972486230efe4bed5e15a63f89ea8937256bc2caafa2265bb1956170771a501"}
Sep 19 15:12:41.247645 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.247625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dssqn" event={"ID":"658ee5aa-e3bf-4397-b6ab-86bdae6793a5","Type":"ContainerStarted","Data":"6bea278fb34a290a65775bef6a20765ba6dbc739853cdb13e3c361cc86350c1c"}
Sep 19 15:12:41.249351 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.249322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth" event={"ID":"da06acaf-ee67-45cd-a1bb-be5e77cf764d","Type":"ContainerStarted","Data":"894ed962c771d6e967c56b3cd2455e9c2c0679e177856ec6a2edc6e11ce8a079"}
Sep 19 15:12:41.252222 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.252207    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:12:41.252579 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.252562    2579 generic.go:356] "Generic (PLEG): container finished" podID="d9d4c00c-9b8d-485f-836c-6af14fe6cd6e" containerID="368366ce560b201889e325691412cfdc0c2b5110b8aa9c8b880e488a68d54f35" exitCode=1
Sep 19 15:12:41.252656 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.252641    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"dacd629dc9f2787e9470b5509bb1310f81782fb53275f74e01bf807647180143"}
Sep 19 15:12:41.252722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.252713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"50db80858ccdf9734f0cce3f8f36c236f1cea2a8d1ef00aa91c27a32bd9aeba0"}
Sep 19 15:12:41.252773 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.252763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"fe90568124a37513d98a9c3eba7cd118c6785aa906de20631d9dabeb0c56bcc5"}
Sep 19 15:12:41.252820 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.252812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"12bab279aa1bd6414a19c67564505333f0444f1f3e07b855d99711db45e5988f"}
Sep 19 15:12:41.252872 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.252859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerDied","Data":"368366ce560b201889e325691412cfdc0c2b5110b8aa9c8b880e488a68d54f35"}
Sep 19 15:12:41.252918 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.252907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"959f9cc37693f7369c8c0c10223f5716cb688caa8d92a70ffe5ffcc85d2223eb"}
Sep 19 15:12:41.254091 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.254072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kk64t" event={"ID":"fbc7f25e-aa96-49eb-b284-4037a2add73b","Type":"ContainerStarted","Data":"a3db837fddc8fd966c029f47a0bb94db7f1053fcf9d1b43fd4a339ef1220e55e"}
Sep 19 15:12:41.254189 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.254057    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d28d4" podStartSLOduration=4.060146362 podStartE2EDuration="21.25404302s" totalImagesPullingTime="17.193896658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:41.253753197 +0000 UTC m=+23.147316837" watchObservedRunningTime="2026-09-19 15:12:41.25404302 +0000 UTC m=+23.147606661"
Sep 19 15:12:41.254189 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.254148    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-0.ec2.internal" podStartSLOduration=20.254138050999998 podStartE2EDuration="20.254138051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:25.224260775 +0000 UTC m=+7.117824416" watchObservedRunningTime="2026-09-19 15:12:41.254138051 +0000 UTC m=+23.147701691"
Sep 19 15:12:41.264993 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.264961    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jz2tb" podStartSLOduration=4.069573719 podStartE2EDuration="21.264950316s" totalImagesPullingTime="17.195376597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:41.264894564 +0000 UTC m=+23.158458202" watchObservedRunningTime="2026-09-19 15:12:41.264950316 +0000 UTC m=+23.158513955"
Sep 19 15:12:41.299343 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.299314    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dssqn" podStartSLOduration=4.060700007 podStartE2EDuration="21.299303177s" totalImagesPullingTime="17.23860317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:41.298912295 +0000 UTC m=+23.192475945" watchObservedRunningTime="2026-09-19 15:12:41.299303177 +0000 UTC m=+23.192866827"
Sep 19 15:12:41.310686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.310652    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7l4fc" podStartSLOduration=11.47941228 podStartE2EDuration="21.310642307s" totalImagesPullingTime="9.831230027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:41.310518106 +0000 UTC m=+23.204081746" watchObservedRunningTime="2026-09-19 15:12:41.310642307 +0000 UTC m=+23.204205943"
Sep 19 15:12:41.323824 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.323793    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kk64t" podStartSLOduration=4.116315693 podStartE2EDuration="21.323782221s" totalImagesPullingTime="17.207466528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:41.32363657 +0000 UTC m=+23.217200209" watchObservedRunningTime="2026-09-19 15:12:41.323782221 +0000 UTC m=+23.217345861"
Sep 19 15:12:41.730733 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:41.730714    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 15:12:42.126245 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:42.126154    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T15:12:41.730729133Z","UUID":"250fc996-40f5-4d6f-b44f-3d6823508ee4","Handler":null,"Name":"","Endpoint":""}
Sep 19 15:12:42.132083 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:42.132056    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 15:12:42.132218 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:42.132099    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 15:12:42.167543 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:42.165701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:42.167543 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:42.165846    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:42.167543 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:42.165885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:42.167543 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:42.165969    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:42.259739 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:42.259704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth" event={"ID":"da06acaf-ee67-45cd-a1bb-be5e77cf764d","Type":"ContainerStarted","Data":"8609567001df2e7e3979e24d37885a642d522556e99209e6db888d3c1cec4170"}
Sep 19 15:12:42.262392 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:42.262365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zwl8d" event={"ID":"5ca6df2a-1979-42bb-80ea-4930186624e9","Type":"ContainerStarted","Data":"d63c11103a044d0fa7b9931fc553b2b9c3e44bc7b7e511999e4455c8b2f7ddb5"}
Sep 19 15:12:42.275551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:42.275498    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zwl8d" podStartSLOduration=5.072430818 podStartE2EDuration="22.275485911s" totalImagesPullingTime="17.203055093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:42.275031671 +0000 UTC m=+24.168595324" watchObservedRunningTime="2026-09-19 15:12:42.275485911 +0000 UTC m=+24.169049551"
Sep 19 15:12:43.164828 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:43.164760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:43.164957 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:43.164911    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:43.266354 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:43.266316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth" event={"ID":"da06acaf-ee67-45cd-a1bb-be5e77cf764d","Type":"ContainerStarted","Data":"399e43f918ac7cce85c059b9075fb68a90648fc30871706ab59e96baac0ca2e1"}
Sep 19 15:12:43.270004 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:43.269976    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:12:43.270409 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:43.270387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"6f16adce0f78757c1d9e60e5f3bb98c14c76d2aaccd3575ef8e0eab767ce2551"}
Sep 19 15:12:43.281688 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:43.281630    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sfsth" podStartSLOduration=4.061222057 podStartE2EDuration="23.281597502s" totalImagesPullingTime="19.220375445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:43.281438288 +0000 UTC m=+25.175001927" watchObservedRunningTime="2026-09-19 15:12:43.281597502 +0000 UTC m=+25.175161161"
Sep 19 15:12:44.164591 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:44.164560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:44.164591 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:44.164593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:44.164825 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:44.164719    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:44.164877 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:44.164842    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:45.165567 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:45.165452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:45.165930 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:45.165687    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:45.279439 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:45.279421    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:12:45.279816 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:45.279795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"0e486fd6b48afb7b72b17e446c23574806d8d9d11660c4f9f1e840150d02a36a"}
Sep 19 15:12:45.280096 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:45.280057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:45.283801 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:45.283779    2579 scope.go:117] "RemoveContainer" containerID="368366ce560b201889e325691412cfdc0c2b5110b8aa9c8b880e488a68d54f35"
Sep 19 15:12:45.297029 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:45.297005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:46.164799 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.164768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:46.164910 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:46.164884    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:46.164962 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.164942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:46.165073 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:46.165055    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:46.172727 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.172703    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:46.173533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.173330    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:46.284864 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.284844    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:12:46.285198 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.285176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" event={"ID":"d9d4c00c-9b8d-485f-836c-6af14fe6cd6e","Type":"ContainerStarted","Data":"c635d7565ec9a7f54dd92765b02239c5871c25456ee6a010bba70fedb858107f"}
Sep 19 15:12:46.285300 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.285259    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 15:12:46.285524 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.285506    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:46.287092 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.287068    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ab9acdc-6a7c-4407-bfdf-85b2f096455f" containerID="971d60f56c354d9994869a76d6ef7a5f219caaf193c4058405cdf266f4ce6891" exitCode=0
Sep 19 15:12:46.287192 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.287145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" event={"ID":"5ab9acdc-6a7c-4407-bfdf-85b2f096455f","Type":"ContainerDied","Data":"971d60f56c354d9994869a76d6ef7a5f219caaf193c4058405cdf266f4ce6891"}
Sep 19 15:12:46.287264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.287249    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:46.287712 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.287697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7l4fc"
Sep 19 15:12:46.301136 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.301120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:46.312676 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:46.311859    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" podStartSLOduration=9.088857442 podStartE2EDuration="26.311842467s" totalImagesPullingTime="17.222985025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:12:46.30923562 +0000 UTC m=+28.202799254" watchObservedRunningTime="2026-09-19 15:12:46.311842467 +0000 UTC m=+28.205406123"
Sep 19 15:12:47.061495 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:47.061298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmfsl"]
Sep 19 15:12:47.061689 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:47.061642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:47.061774 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:47.061750    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:47.063885 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:47.063862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4mr5z"]
Sep 19 15:12:47.064012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:47.063953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:47.064068 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:47.064037    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:47.073898 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:47.073799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vfsx"]
Sep 19 15:12:47.073898 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:47.073883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:47.074035 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:47.073960    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:47.289212 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:47.289194    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 15:12:48.292877 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:48.292841    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ab9acdc-6a7c-4407-bfdf-85b2f096455f" containerID="2601f97d57d3e869ea8fa07ffd11444268c79dcf6dcf45ddb0803e53c4519659" exitCode=0
Sep 19 15:12:48.293243 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:48.292916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" event={"ID":"5ab9acdc-6a7c-4407-bfdf-85b2f096455f","Type":"ContainerDied","Data":"2601f97d57d3e869ea8fa07ffd11444268c79dcf6dcf45ddb0803e53c4519659"}
Sep 19 15:12:48.293243 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:48.292991    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 15:12:49.165408 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:49.165385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:49.165522 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:49.165476    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:49.165522 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:49.165385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:49.165522 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:49.165510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:49.165694 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:49.165628    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:49.165729 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:49.165714    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:50.234732 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:50.234705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:12:50.235078 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:50.234864    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 15:12:50.249936 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:50.249889    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" podUID="d9d4c00c-9b8d-485f-836c-6af14fe6cd6e" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 19 15:12:50.258291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:50.258260    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77" podUID="d9d4c00c-9b8d-485f-836c-6af14fe6cd6e" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 19 15:12:50.298539 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:50.298510    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ab9acdc-6a7c-4407-bfdf-85b2f096455f" containerID="d9c327fb0d872aa865553394fc6a509e9fa097aba5ca09b1ce15da026bf0af4e" exitCode=0
Sep 19 15:12:50.298643 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:50.298575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" event={"ID":"5ab9acdc-6a7c-4407-bfdf-85b2f096455f","Type":"ContainerDied","Data":"d9c327fb0d872aa865553394fc6a509e9fa097aba5ca09b1ce15da026bf0af4e"}
Sep 19 15:12:51.164867 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:51.164833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:51.164867 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:51.164857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:51.165056 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:51.164941    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4mr5z" podUID="ff785d9b-cbc1-48e2-9af1-53f35353cd9a"
Sep 19 15:12:51.165056 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:51.164993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:51.165136 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:51.165066    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmfsl" podUID="d9e06119-2d4a-42ce-b018-9f947d18ad98"
Sep 19 15:12:51.165170 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:51.165137    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:12:53.090316 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.090037    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-0.ec2.internal" event="NodeReady"
Sep 19 15:12:53.090738 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.090396    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 15:12:53.129764 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.129734    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z2p7w"]
Sep 19 15:12:53.156058 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.156027    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rgzxt"]
Sep 19 15:12:53.156209 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.156170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.158783 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.158756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 15:12:53.158905 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.158868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p4wjr\""
Sep 19 15:12:53.159041 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.159019    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 15:12:53.175907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.175887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z2p7w"]
Sep 19 15:12:53.176032 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.175919    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rgzxt"]
Sep 19 15:12:53.176032 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.175986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:53.176137 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.175984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:53.176237 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.175984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rgzxt"
Sep 19 15:12:53.176368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.175986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.177915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.177952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.178033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.178307    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.178343    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zprn6\""
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.178359    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v2755\""
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.178426    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxxj8\""
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.178307    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 15:12:53.178510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.178345    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 15:12:53.179071 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.178692    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 15:12:53.311365 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.311327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fcfea47-b51c-41cb-90b0-e656bd0640d5-cert\") pod \"ingress-canary-rgzxt\" (UID: \"7fcfea47-b51c-41cb-90b0-e656bd0640d5\") " pod="openshift-ingress-canary/ingress-canary-rgzxt"
Sep 19 15:12:53.311529 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.311435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2dc993-51e2-4234-a738-9dd1d61891df-tmp-dir\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.311529 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.311476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.311529 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.311504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4np\" (UniqueName: \"kubernetes.io/projected/bb2dc993-51e2-4234-a738-9dd1d61891df-kube-api-access-hw4np\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.311666 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.311552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbd4w\" (UniqueName: \"kubernetes.io/projected/7fcfea47-b51c-41cb-90b0-e656bd0640d5-kube-api-access-wbd4w\") pod \"ingress-canary-rgzxt\" (UID: \"7fcfea47-b51c-41cb-90b0-e656bd0640d5\") " pod="openshift-ingress-canary/ingress-canary-rgzxt"
Sep 19 15:12:53.311666 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.311579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2dc993-51e2-4234-a738-9dd1d61891df-config-volume\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.412262 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.412182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2dc993-51e2-4234-a738-9dd1d61891df-tmp-dir\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.412262 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.412220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.412262 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.412250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4np\" (UniqueName: \"kubernetes.io/projected/bb2dc993-51e2-4234-a738-9dd1d61891df-kube-api-access-hw4np\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.412510 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:53.412365    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:12:53.412510 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:53.412451    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:12:53.912428887 +0000 UTC m=+35.805992508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:12:53.412639 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.412586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbd4w\" (UniqueName: \"kubernetes.io/projected/7fcfea47-b51c-41cb-90b0-e656bd0640d5-kube-api-access-wbd4w\") pod \"ingress-canary-rgzxt\" (UID: \"7fcfea47-b51c-41cb-90b0-e656bd0640d5\") " pod="openshift-ingress-canary/ingress-canary-rgzxt"
Sep 19 15:12:53.412703 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.412642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2dc993-51e2-4234-a738-9dd1d61891df-config-volume\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.412703 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.412648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2dc993-51e2-4234-a738-9dd1d61891df-tmp-dir\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.412703 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.412668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fcfea47-b51c-41cb-90b0-e656bd0640d5-cert\") pod \"ingress-canary-rgzxt\" (UID: \"7fcfea47-b51c-41cb-90b0-e656bd0640d5\") " pod="openshift-ingress-canary/ingress-canary-rgzxt"
Sep 19 15:12:53.413169 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.413146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb2dc993-51e2-4234-a738-9dd1d61891df-config-volume\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.417224 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.417195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fcfea47-b51c-41cb-90b0-e656bd0640d5-cert\") pod \"ingress-canary-rgzxt\" (UID: \"7fcfea47-b51c-41cb-90b0-e656bd0640d5\") " pod="openshift-ingress-canary/ingress-canary-rgzxt"
Sep 19 15:12:53.422798 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.422777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4np\" (UniqueName: \"kubernetes.io/projected/bb2dc993-51e2-4234-a738-9dd1d61891df-kube-api-access-hw4np\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.423252 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.423237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbd4w\" (UniqueName: \"kubernetes.io/projected/7fcfea47-b51c-41cb-90b0-e656bd0640d5-kube-api-access-wbd4w\") pod \"ingress-canary-rgzxt\" (UID: \"7fcfea47-b51c-41cb-90b0-e656bd0640d5\") " pod="openshift-ingress-canary/ingress-canary-rgzxt"
Sep 19 15:12:53.499055 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.499023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rgzxt"
Sep 19 15:12:53.815567 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.815532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:53.815791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.815614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:12:53.815791 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:53.815695    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 15:12:53.815791 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:53.815762    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:13:25.815744575 +0000 UTC m=+67.709308199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : secret "metrics-daemon-secret" not found
Sep 19 15:12:53.818141 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.818117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9e06119-2d4a-42ce-b018-9f947d18ad98-original-pull-secret\") pod \"global-pull-secret-syncer-zmfsl\" (UID: \"d9e06119-2d4a-42ce-b018-9f947d18ad98\") " pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:53.916214 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.916175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:53.916386 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.916239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:53.916386 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:53.916297    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:12:53.916386 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:53.916369    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:12:54.91634873 +0000 UTC m=+36.809912353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:12:53.919029 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:53.919003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dscp\" (UniqueName: \"kubernetes.io/projected/ff785d9b-cbc1-48e2-9af1-53f35353cd9a-kube-api-access-8dscp\") pod \"network-check-target-4mr5z\" (UID: \"ff785d9b-cbc1-48e2-9af1-53f35353cd9a\") " pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:54.086962 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:54.086881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmfsl"
Sep 19 15:12:54.105278 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:54.105257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:12:54.614336 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:54.614298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmfsl"]
Sep 19 15:12:54.617382 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:54.617352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4mr5z"]
Sep 19 15:12:54.617978 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:54.617956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rgzxt"]
Sep 19 15:12:54.631905 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:54.631877    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e06119_2d4a_42ce_b018_9f947d18ad98.slice/crio-1f999db4da4a6e784badf2b13d774e8ba31a84d305fff0c772c79a3acf2e97c1 WatchSource:0}: Error finding container 1f999db4da4a6e784badf2b13d774e8ba31a84d305fff0c772c79a3acf2e97c1: Status 404 returned error can't find the container with id 1f999db4da4a6e784badf2b13d774e8ba31a84d305fff0c772c79a3acf2e97c1
Sep 19 15:12:54.632162 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:54.632139    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff785d9b_cbc1_48e2_9af1_53f35353cd9a.slice/crio-3852eebb6336a691e936c4bcf1e017ba04b5e6a8afbceeb88e467f64b1f6d376 WatchSource:0}: Error finding container 3852eebb6336a691e936c4bcf1e017ba04b5e6a8afbceeb88e467f64b1f6d376: Status 404 returned error can't find the container with id 3852eebb6336a691e936c4bcf1e017ba04b5e6a8afbceeb88e467f64b1f6d376
Sep 19 15:12:54.632650 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:12:54.632629    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfea47_b51c_41cb_90b0_e656bd0640d5.slice/crio-818b059b744cb3b3330298abe650fcf7c0e02d0cd382493ba8e70b8c4ac47322 WatchSource:0}: Error finding container 818b059b744cb3b3330298abe650fcf7c0e02d0cd382493ba8e70b8c4ac47322: Status 404 returned error can't find the container with id 818b059b744cb3b3330298abe650fcf7c0e02d0cd382493ba8e70b8c4ac47322
Sep 19 15:12:54.922863 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:54.922685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:54.922958 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:54.922855    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:12:54.922958 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:54.922945    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:12:56.922922943 +0000 UTC m=+38.816486565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:12:55.315253 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:55.315216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4mr5z" event={"ID":"ff785d9b-cbc1-48e2-9af1-53f35353cd9a","Type":"ContainerStarted","Data":"3852eebb6336a691e936c4bcf1e017ba04b5e6a8afbceeb88e467f64b1f6d376"}
Sep 19 15:12:55.319759 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:55.318933    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ab9acdc-6a7c-4407-bfdf-85b2f096455f" containerID="1cf629decfc42f4a73fdb62312809a34c0202129aee8b734ec86c29bb988d1a5" exitCode=0
Sep 19 15:12:55.319759 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:55.319018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" event={"ID":"5ab9acdc-6a7c-4407-bfdf-85b2f096455f","Type":"ContainerDied","Data":"1cf629decfc42f4a73fdb62312809a34c0202129aee8b734ec86c29bb988d1a5"}
Sep 19 15:12:55.323594 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:55.321849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmfsl" event={"ID":"d9e06119-2d4a-42ce-b018-9f947d18ad98","Type":"ContainerStarted","Data":"1f999db4da4a6e784badf2b13d774e8ba31a84d305fff0c772c79a3acf2e97c1"}
Sep 19 15:12:55.323594 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:55.323305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rgzxt" event={"ID":"7fcfea47-b51c-41cb-90b0-e656bd0640d5","Type":"ContainerStarted","Data":"818b059b744cb3b3330298abe650fcf7c0e02d0cd382493ba8e70b8c4ac47322"}
Sep 19 15:12:56.331456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:56.331417    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ab9acdc-6a7c-4407-bfdf-85b2f096455f" containerID="f503128a7f6d0ee7e7ac7abbd147e02e1ea1b8b696e1a5cd4b34dd525398b8dc" exitCode=0
Sep 19 15:12:56.332007 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:56.331504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" event={"ID":"5ab9acdc-6a7c-4407-bfdf-85b2f096455f","Type":"ContainerDied","Data":"f503128a7f6d0ee7e7ac7abbd147e02e1ea1b8b696e1a5cd4b34dd525398b8dc"}
Sep 19 15:12:56.939100 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:12:56.939066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:12:56.939292 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:56.939213    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:12:56.939292 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:12:56.939277    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:13:00.939258401 +0000 UTC m=+42.832822025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:13:00.344317 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.344277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" event={"ID":"5ab9acdc-6a7c-4407-bfdf-85b2f096455f","Type":"ContainerStarted","Data":"57c6ed0e75d0b475fed31dc348ede6a89fe142c1b4297c5651197c9334178c42"}
Sep 19 15:13:00.345705 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.345682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmfsl" event={"ID":"d9e06119-2d4a-42ce-b018-9f947d18ad98","Type":"ContainerStarted","Data":"44aeb82fc3772c4f82b01c3d077df9204eb4e5d897a8294631f7ead1542b617f"}
Sep 19 15:13:00.347054 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.347031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rgzxt" event={"ID":"7fcfea47-b51c-41cb-90b0-e656bd0640d5","Type":"ContainerStarted","Data":"ec0c9214e93333d0d61ecde9346437f8801127da4c1bd65c21b4a8b730148355"}
Sep 19 15:13:00.348508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.348491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4mr5z" event={"ID":"ff785d9b-cbc1-48e2-9af1-53f35353cd9a","Type":"ContainerStarted","Data":"e823ff420eb6061d3d6f18390f2335b13661b16c0607d39a59efec86325854d2"}
Sep 19 15:13:00.348624 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.348618    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:13:00.363440 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.363396    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8t2m7" podStartSLOduration=12.825977017 podStartE2EDuration="40.363380076s" totalImagesPullingTime="27.537403059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:13:00.362189697 +0000 UTC m=+42.255753337" watchObservedRunningTime="2026-09-19 15:13:00.363380076 +0000 UTC m=+42.256943717"
Sep 19 15:13:00.374804 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.374770    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rgzxt" podStartSLOduration=2.4565353659999998 podStartE2EDuration="7.374753996s" totalImagesPullingTime="4.91821863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:13:00.374289648 +0000 UTC m=+42.267853288" watchObservedRunningTime="2026-09-19 15:13:00.374753996 +0000 UTC m=+42.268317636"
Sep 19 15:13:00.386654 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.386622    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zmfsl" podStartSLOduration=35.457791739 podStartE2EDuration="40.386596186s" totalImagesPullingTime="4.928804447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:13:00.385821317 +0000 UTC m=+42.279384984" watchObservedRunningTime="2026-09-19 15:13:00.386596186 +0000 UTC m=+42.280159848"
Sep 19 15:13:00.397907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.397877    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4mr5z" podStartSLOduration=35.467068777 podStartE2EDuration="40.397866586s" totalImagesPullingTime="4.930797809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:13:00.397327819 +0000 UTC m=+42.290891458" watchObservedRunningTime="2026-09-19 15:13:00.397866586 +0000 UTC m=+42.291430225"
Sep 19 15:13:00.968308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:00.968279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:13:00.968468 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:00.968388    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:13:00.968468 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:00.968435    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:13:08.968423333 +0000 UTC m=+50.861986952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:13:09.014472 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:09.014438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:13:09.014852 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:09.014541    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:13:09.014852 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:09.014592    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:13:25.014579751 +0000 UTC m=+66.908143369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:13:20.259906 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:20.259881    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5v77"
Sep 19 15:13:25.110768 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:25.110732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:13:25.111143 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:25.110838    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:13:25.111143 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:25.110926    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:13:57.110912607 +0000 UTC m=+99.004476225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:13:25.914301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:25.914268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:13:25.914517 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:25.914369    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 15:13:25.914517 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:25.914422    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:14:29.914408581 +0000 UTC m=+131.807972199 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : secret "metrics-daemon-secret" not found
Sep 19 15:13:26.139796 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.139769    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"]
Sep 19 15:13:26.142610 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.142585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.144656 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.144633    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 15:13:26.144656 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.144639    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 19 15:13:26.144830 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.144672    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 19 15:13:26.145478 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.145456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 19 15:13:26.145715 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.145693    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 19 15:13:26.145715 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.145710    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 15:13:26.145853 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.145820    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 15:13:26.152437 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.152417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"]
Sep 19 15:13:26.316361 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.316334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.316530 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.316367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksv2m\" (UniqueName: \"kubernetes.io/projected/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-kube-api-access-ksv2m\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.316530 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.316394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.316530 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.316465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-ca\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.316530 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.316494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.316682 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.316540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-hub\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.417184 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.417159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-ca\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.417291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.417191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.417291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.417224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-hub\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.417291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.417240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.417291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.417263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksv2m\" (UniqueName: \"kubernetes.io/projected/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-kube-api-access-ksv2m\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.417291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.417287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.417978 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.417949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.420649 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.420624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-ca\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.420730 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.420684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.420972 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.420957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.421020 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.421000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-hub\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.424112 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.424091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksv2m\" (UniqueName: \"kubernetes.io/projected/91c52b5e-06d8-4c9c-adfb-7da8f962f48c-kube-api-access-ksv2m\") pod \"cluster-proxy-proxy-agent-767966c968-ghgfr\" (UID: \"91c52b5e-06d8-4c9c-adfb-7da8f962f48c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.459779 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.459759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:13:26.723850 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:26.723819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"]
Sep 19 15:13:26.727183 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:13:26.727158    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91c52b5e_06d8_4c9c_adfb_7da8f962f48c.slice/crio-e94042a03910196c435947911398af6af647420273937c331db6e0513ef7a231 WatchSource:0}: Error finding container e94042a03910196c435947911398af6af647420273937c331db6e0513ef7a231: Status 404 returned error can't find the container with id e94042a03910196c435947911398af6af647420273937c331db6e0513ef7a231
Sep 19 15:13:27.411333 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:27.411297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" event={"ID":"91c52b5e-06d8-4c9c-adfb-7da8f962f48c","Type":"ContainerStarted","Data":"e94042a03910196c435947911398af6af647420273937c331db6e0513ef7a231"}
Sep 19 15:13:31.353787 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:31.353676    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4mr5z"
Sep 19 15:13:53.477356 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:53.477322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" event={"ID":"91c52b5e-06d8-4c9c-adfb-7da8f962f48c","Type":"ContainerStarted","Data":"db05992f18587665ee3c2d080b6a8cfd785e50df3a81d4ff5d1d197449393a89"}
Sep 19 15:13:53.477356 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:53.477362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" event={"ID":"91c52b5e-06d8-4c9c-adfb-7da8f962f48c","Type":"ContainerStarted","Data":"9b247a1830531af05d4a015a7e5ff98f7aba5256c72dca2f174a89c532229553"}
Sep 19 15:13:53.477920 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:53.477373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" event={"ID":"91c52b5e-06d8-4c9c-adfb-7da8f962f48c","Type":"ContainerStarted","Data":"30df4ee1efdd5b201b77b6438459f5b9a2f3525f19dd2bdcdcfd554ba7897f31"}
Sep 19 15:13:53.494298 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:53.494236    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" podStartSLOduration=1.703446002 podStartE2EDuration="27.494204018s" totalImagesPullingTime="25.790758016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:13:53.494176687 +0000 UTC m=+95.387740327" watchObservedRunningTime="2026-09-19 15:13:53.494204018 +0000 UTC m=+95.387767636"
Sep 19 15:13:57.209319 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:13:57.209284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:13:57.209718 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:57.209388    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:13:57.209718 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:13:57.209450    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:15:01.209437325 +0000 UTC m=+163.103000943 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:14:29.915456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:14:29.915415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:14:29.915942 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:14:29.915561    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 15:14:29.915942 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:14:29.915651    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs podName:e261dff5-21eb-4a06-9996-349666d1fd9f nodeName:}" failed. No retries permitted until 2026-09-19 15:16:31.915634972 +0000 UTC m=+253.809198593 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs") pod "network-metrics-daemon-7vfsx" (UID: "e261dff5-21eb-4a06-9996-349666d1fd9f") : secret "metrics-daemon-secret" not found
Sep 19 15:14:56.168027 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:14:56.167992    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z2p7w" podUID="bb2dc993-51e2-4234-a738-9dd1d61891df"
Sep 19 15:14:56.193869 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:14:56.193836    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7vfsx" podUID="e261dff5-21eb-4a06-9996-349666d1fd9f"
Sep 19 15:14:56.647391 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:14:56.647365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:15:01.214534 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:15:01.214444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:15:01.214925 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:15:01.214583    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 15:15:01.214925 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:15:01.214662    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls podName:bb2dc993-51e2-4234-a738-9dd1d61891df nodeName:}" failed. No retries permitted until 2026-09-19 15:17:03.214645871 +0000 UTC m=+285.108209502 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls") pod "dns-default-z2p7w" (UID: "bb2dc993-51e2-4234-a738-9dd1d61891df") : secret "dns-default-metrics-tls" not found
Sep 19 15:15:07.164557 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:15:07.164522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:16:19.875966 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.875914    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ssz8p"]
Sep 19 15:16:19.878826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.878811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:19.880763 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.880738    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 15:16:19.880916 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.880900    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 15:16:19.880990 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.880975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 15:16:19.881051 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.881039    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 15:16:19.881624 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.881590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkfcq\""
Sep 19 15:16:19.887645 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.887625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ssz8p"]
Sep 19 15:16:19.928286 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.928261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-crio-socket\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:19.928373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.928294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjwk\" (UniqueName: \"kubernetes.io/projected/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-api-access-7wjwk\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:19.928373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.928313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-data-volume\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:19.928373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.928348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:19.928471 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:19.928389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.029146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.029124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-crio-socket\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.029219 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.029155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjwk\" (UniqueName: \"kubernetes.io/projected/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-api-access-7wjwk\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.029219 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.029180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-data-volume\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.029219 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.029210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.029344 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.029225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.029344 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.029241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-crio-socket\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.029564 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.029549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-data-volume\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.030833 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.030810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 15:16:20.030875 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.030855    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 15:16:20.035046 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.035032    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 15:16:20.039454 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:20.039432    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 15:16:20.039508 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:20.039442    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:20.039541 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:20.039511    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:20.539497632 +0000 UTC m=+242.433061263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : configmap "kube-rbac-proxy" not found
Sep 19 15:16:20.039579 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:20.039543    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:20.539530351 +0000 UTC m=+242.433093973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:20.045942 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.045929    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 15:16:20.057170 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.057148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjwk\" (UniqueName: \"kubernetes.io/projected/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-api-access-7wjwk\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.634335 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.634306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.634335 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:20.634336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:20.634523 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:20.634407    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 15:16:20.634523 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:20.634445    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:20.634523 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:20.634454    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:21.634442223 +0000 UTC m=+243.528005841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : configmap "kube-rbac-proxy" not found
Sep 19 15:16:20.634523 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:20.634513    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:21.634496135 +0000 UTC m=+243.528059757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:21.022809 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.022778    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-fb4mr"]
Sep 19 15:16:21.025668 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.025648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.028205 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.028181    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 19 15:16:21.028205 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.028195    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 19 15:16:21.028205 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.028208    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-slxfq\""
Sep 19 15:16:21.028402 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.028184    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 19 15:16:21.028402 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.028274    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 19 15:16:21.032322 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.032301    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-fb4mr"]
Sep 19 15:16:21.138012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.137984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/398432f7-dcdb-4f65-be80-4e2b037ba1ab-signing-cabundle\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.138124 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.138023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrht\" (UniqueName: \"kubernetes.io/projected/398432f7-dcdb-4f65-be80-4e2b037ba1ab-kube-api-access-hnrht\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.138164 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.138124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/398432f7-dcdb-4f65-be80-4e2b037ba1ab-signing-key\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.239342 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.239312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/398432f7-dcdb-4f65-be80-4e2b037ba1ab-signing-key\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.239460 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.239368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/398432f7-dcdb-4f65-be80-4e2b037ba1ab-signing-cabundle\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.239516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.239495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrht\" (UniqueName: \"kubernetes.io/projected/398432f7-dcdb-4f65-be80-4e2b037ba1ab-kube-api-access-hnrht\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.240059 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.240043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/398432f7-dcdb-4f65-be80-4e2b037ba1ab-signing-cabundle\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.241851 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.241824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/398432f7-dcdb-4f65-be80-4e2b037ba1ab-signing-key\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.246551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.246530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrht\" (UniqueName: \"kubernetes.io/projected/398432f7-dcdb-4f65-be80-4e2b037ba1ab-kube-api-access-hnrht\") pod \"service-ca-55f6fc7b86-fb4mr\" (UID: \"398432f7-dcdb-4f65-be80-4e2b037ba1ab\") " pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.334864 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.334813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr"
Sep 19 15:16:21.600406 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.600328    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-fb4mr"]
Sep 19 15:16:21.603612 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:16:21.603575    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398432f7_dcdb_4f65_be80_4e2b037ba1ab.slice/crio-19406e5f69aa86b628f8282c016e4f958c6c0922025aafc3d8c502d6db0ede20 WatchSource:0}: Error finding container 19406e5f69aa86b628f8282c016e4f958c6c0922025aafc3d8c502d6db0ede20: Status 404 returned error can't find the container with id 19406e5f69aa86b628f8282c016e4f958c6c0922025aafc3d8c502d6db0ede20
Sep 19 15:16:21.643024 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.643002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:21.643127 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.643034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:21.643165 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:21.643138    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 15:16:21.643165 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:21.643147    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:21.643228 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:21.643192    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:23.643175887 +0000 UTC m=+245.536739521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : configmap "kube-rbac-proxy" not found
Sep 19 15:16:21.643228 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:21.643205    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:23.643198803 +0000 UTC m=+245.536762421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:21.880145 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:21.880075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr" event={"ID":"398432f7-dcdb-4f65-be80-4e2b037ba1ab","Type":"ContainerStarted","Data":"19406e5f69aa86b628f8282c016e4f958c6c0922025aafc3d8c502d6db0ede20"}
Sep 19 15:16:22.643355 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:22.643324    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jz2tb_b9ecfb66-d607-4989-8f1e-3651e26cc2a3/dns-node-resolver/0.log"
Sep 19 15:16:23.442902 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:23.442881    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d28d4_500406fa-6c94-45a7-a391-5c758bdd727b/node-ca/0.log"
Sep 19 15:16:23.659275 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:23.659242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:23.659275 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:23.659281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:23.659961 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:23.659354    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 15:16:23.659961 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:23.659389    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:23.659961 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:23.659429    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:27.659414972 +0000 UTC m=+249.552978591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : configmap "kube-rbac-proxy" not found
Sep 19 15:16:23.659961 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:23.659442    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:27.65943638 +0000 UTC m=+249.552999998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:23.889310 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:23.889281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr" event={"ID":"398432f7-dcdb-4f65-be80-4e2b037ba1ab","Type":"ContainerStarted","Data":"11c82ac7865bb4a4067b9647c525158d86b62b866cd950144b549d6ba31e78dc"}
Sep 19 15:16:23.904001 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:23.903956    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-fb4mr" podStartSLOduration=1.093022041 podStartE2EDuration="2.903941893s" totalImagesPullingTime="1.810919852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:16:23.903257077 +0000 UTC m=+245.796820715" watchObservedRunningTime="2026-09-19 15:16:23.903941893 +0000 UTC m=+245.797505532"
Sep 19 15:16:24.444468 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:24.444392    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rgzxt_7fcfea47-b51c-41cb-90b0-e656bd0640d5/serve-healthcheck-canary/0.log"
Sep 19 15:16:27.686874 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:27.686837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:27.686874 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:27.686877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:27.687482 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:27.686963    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 15:16:27.687482 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:27.686995    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:27.687482 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:27.687045    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:35.687026968 +0000 UTC m=+257.580590589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : configmap "kube-rbac-proxy" not found
Sep 19 15:16:27.687482 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:27.687057    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls podName:b7b96ff1-119d-4684-8be3-c5e2794dc6c1 nodeName:}" failed. No retries permitted until 2026-09-19 15:16:35.687052093 +0000 UTC m=+257.580615711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ssz8p" (UID: "b7b96ff1-119d-4684-8be3-c5e2794dc6c1") : secret "insights-runtime-extractor-tls" not found
Sep 19 15:16:31.918061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:31.917980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:16:31.920526 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:31.920495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e261dff5-21eb-4a06-9996-349666d1fd9f-metrics-certs\") pod \"network-metrics-daemon-7vfsx\" (UID: \"e261dff5-21eb-4a06-9996-349666d1fd9f\") " pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:16:32.067438 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:32.067409    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxxj8\""
Sep 19 15:16:32.075889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:32.075876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vfsx"
Sep 19 15:16:32.342355 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:32.342321    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vfsx"]
Sep 19 15:16:32.345735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:16:32.345705    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode261dff5_21eb_4a06_9996_349666d1fd9f.slice/crio-d4936a6418184c85ef83e6cdab341f319dc3d8655e7f7d6370790b053b2915ce WatchSource:0}: Error finding container d4936a6418184c85ef83e6cdab341f319dc3d8655e7f7d6370790b053b2915ce: Status 404 returned error can't find the container with id d4936a6418184c85ef83e6cdab341f319dc3d8655e7f7d6370790b053b2915ce
Sep 19 15:16:32.918983 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:32.918940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vfsx" event={"ID":"e261dff5-21eb-4a06-9996-349666d1fd9f","Type":"ContainerStarted","Data":"d4936a6418184c85ef83e6cdab341f319dc3d8655e7f7d6370790b053b2915ce"}
Sep 19 15:16:33.923298 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:33.923264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vfsx" event={"ID":"e261dff5-21eb-4a06-9996-349666d1fd9f","Type":"ContainerStarted","Data":"f516a7de68fe31b4743e1be7a962fbbf3322d4b975259bc852e5f006fd31f1f8"}
Sep 19 15:16:33.923696 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:33.923313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vfsx" event={"ID":"e261dff5-21eb-4a06-9996-349666d1fd9f","Type":"ContainerStarted","Data":"c7ecc783ef902421ff42254ddbce65da3abe9c7506a98e1f0d112261e00f87c2"}
Sep 19 15:16:33.936490 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:33.936441    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7vfsx" podStartSLOduration=253.014003608 podStartE2EDuration="4m13.936428085s" totalImagesPullingTime="922.424477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:16:33.935536587 +0000 UTC m=+255.829100237" watchObservedRunningTime="2026-09-19 15:16:33.936428085 +0000 UTC m=+255.829991724"
Sep 19 15:16:35.745476 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:35.745443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:35.745476 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:35.745479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:35.745950 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:35.745933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:35.747863 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:35.747841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7b96ff1-119d-4684-8be3-c5e2794dc6c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ssz8p\" (UID: \"b7b96ff1-119d-4684-8be3-c5e2794dc6c1\") " pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:35.789755 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:35.789731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkfcq\""
Sep 19 15:16:35.798446 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:35.798430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ssz8p"
Sep 19 15:16:36.063448 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:36.063382    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ssz8p"]
Sep 19 15:16:36.065952 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:16:36.065927    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b96ff1_119d_4684_8be3_c5e2794dc6c1.slice/crio-581d554c1a0a1c272a15fe9d746940b426cf1c0c075ccd1ed935f9fb8f2b3b27 WatchSource:0}: Error finding container 581d554c1a0a1c272a15fe9d746940b426cf1c0c075ccd1ed935f9fb8f2b3b27: Status 404 returned error can't find the container with id 581d554c1a0a1c272a15fe9d746940b426cf1c0c075ccd1ed935f9fb8f2b3b27
Sep 19 15:16:36.933867 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:36.933831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ssz8p" event={"ID":"b7b96ff1-119d-4684-8be3-c5e2794dc6c1","Type":"ContainerStarted","Data":"407978fb3e3a174363a8ebda388af448eed13a58c7d9497db42c8b0e6f0d1038"}
Sep 19 15:16:36.934163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:36.933911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ssz8p" event={"ID":"b7b96ff1-119d-4684-8be3-c5e2794dc6c1","Type":"ContainerStarted","Data":"0826704566ab742ecea729c068ae3b05171943c4a20d6d561bfa79f6504846b6"}
Sep 19 15:16:36.934163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:36.933930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ssz8p" event={"ID":"b7b96ff1-119d-4684-8be3-c5e2794dc6c1","Type":"ContainerStarted","Data":"581d554c1a0a1c272a15fe9d746940b426cf1c0c075ccd1ed935f9fb8f2b3b27"}
Sep 19 15:16:38.941704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:38.941669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ssz8p" event={"ID":"b7b96ff1-119d-4684-8be3-c5e2794dc6c1","Type":"ContainerStarted","Data":"563dac7dee55769fa0b67e9ae1e840742650dff51be2efbaa1c9344f4d3b5dbb"}
Sep 19 15:16:38.958826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:38.958781    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ssz8p" podStartSLOduration=18.030388427 podStartE2EDuration="19.958763293s" totalImagesPullingTime="1.928374866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:16:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:16:38.957730259 +0000 UTC m=+260.851293900" watchObservedRunningTime="2026-09-19 15:16:38.958763293 +0000 UTC m=+260.852326932"
Sep 19 15:16:40.196795 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.196765    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64d48d969c-vl59s"]
Sep 19 15:16:40.199551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.199536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.202057 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.202035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 15:16:40.202188 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.202063    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 15:16:40.202666 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.202650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 15:16:40.202763 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.202651    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xbxpb\""
Sep 19 15:16:40.207468 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.207447    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 15:16:40.212169 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.212147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64d48d969c-vl59s"]
Sep 19 15:16:40.277157 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.277132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-bound-sa-token\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.277261 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.277160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-registry-tls\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.277261 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.277176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3436924-04a9-449d-8276-4d71763093dc-ca-trust-extracted\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.277261 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.277198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3436924-04a9-449d-8276-4d71763093dc-installation-pull-secrets\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.277370 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.277275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghg5\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-kube-api-access-2ghg5\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.277370 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.277299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3436924-04a9-449d-8276-4d71763093dc-trusted-ca\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.277370 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.277324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3436924-04a9-449d-8276-4d71763093dc-image-registry-private-configuration\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.277370 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.277340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3436924-04a9-449d-8276-4d71763093dc-registry-certificates\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-bound-sa-token\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-registry-tls\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3436924-04a9-449d-8276-4d71763093dc-ca-trust-extracted\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3436924-04a9-449d-8276-4d71763093dc-installation-pull-secrets\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghg5\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-kube-api-access-2ghg5\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378315 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3436924-04a9-449d-8276-4d71763093dc-trusted-ca\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3436924-04a9-449d-8276-4d71763093dc-image-registry-private-configuration\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378658 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3436924-04a9-449d-8276-4d71763093dc-registry-certificates\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.378826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.378700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3436924-04a9-449d-8276-4d71763093dc-ca-trust-extracted\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.379327 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.379308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3436924-04a9-449d-8276-4d71763093dc-trusted-ca\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.379819 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.379795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3436924-04a9-449d-8276-4d71763093dc-registry-certificates\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.380772 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.380748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3436924-04a9-449d-8276-4d71763093dc-installation-pull-secrets\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.380903 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.380887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-registry-tls\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.380962 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.380926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3436924-04a9-449d-8276-4d71763093dc-image-registry-private-configuration\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.385543 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.385521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-bound-sa-token\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.386826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.386806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghg5\" (UniqueName: \"kubernetes.io/projected/e3436924-04a9-449d-8276-4d71763093dc-kube-api-access-2ghg5\") pod \"image-registry-64d48d969c-vl59s\" (UID: \"e3436924-04a9-449d-8276-4d71763093dc\") " pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.507851 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.507791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.775100 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.775030    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64d48d969c-vl59s"]
Sep 19 15:16:40.778424 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:16:40.778401    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3436924_04a9_449d_8276_4d71763093dc.slice/crio-8c9f610b8a428c3f9704f1dd43aeac97ca184150da0f0573851b4ce3db2fe27e WatchSource:0}: Error finding container 8c9f610b8a428c3f9704f1dd43aeac97ca184150da0f0573851b4ce3db2fe27e: Status 404 returned error can't find the container with id 8c9f610b8a428c3f9704f1dd43aeac97ca184150da0f0573851b4ce3db2fe27e
Sep 19 15:16:40.949176 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.949133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64d48d969c-vl59s" event={"ID":"e3436924-04a9-449d-8276-4d71763093dc","Type":"ContainerStarted","Data":"2eb3ff443a2d7c6e8ebcb14e6711ce7533c36bad4950d92e65db820172b9470f"}
Sep 19 15:16:40.949325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.949195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64d48d969c-vl59s" event={"ID":"e3436924-04a9-449d-8276-4d71763093dc","Type":"ContainerStarted","Data":"8c9f610b8a428c3f9704f1dd43aeac97ca184150da0f0573851b4ce3db2fe27e"}
Sep 19 15:16:40.949325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.949237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:16:40.969191 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:16:40.969148    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64d48d969c-vl59s" podStartSLOduration=0.969131661 podStartE2EDuration="969.131661ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:16:40.968428637 +0000 UTC m=+262.861992307" watchObservedRunningTime="2026-09-19 15:16:40.969131661 +0000 UTC m=+262.862695311"
Sep 19 15:16:59.647957 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:16:59.647917    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z2p7w" podUID="bb2dc993-51e2-4234-a738-9dd1d61891df"
Sep 19 15:17:00.004831 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:00.004803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:17:00.512373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:00.512346    2579 patch_prober.go:28] interesting pod/image-registry-64d48d969c-vl59s container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 19 15:17:00.512516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:00.512395    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64d48d969c-vl59s" podUID="e3436924-04a9-449d-8276-4d71763093dc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 15:17:01.956577 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:01.956548    2579 patch_prober.go:28] interesting pod/image-registry-64d48d969c-vl59s container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 19 15:17:01.956948 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:01.956594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64d48d969c-vl59s" podUID="e3436924-04a9-449d-8276-4d71763093dc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 15:17:03.222983 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:03.222950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:17:03.225340 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:03.225315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb2dc993-51e2-4234-a738-9dd1d61891df-metrics-tls\") pod \"dns-default-z2p7w\" (UID: \"bb2dc993-51e2-4234-a738-9dd1d61891df\") " pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:17:03.307907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:03.307885    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p4wjr\""
Sep 19 15:17:03.316506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:03.316485    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:17:03.582618 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:03.582525    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z2p7w"]
Sep 19 15:17:03.587696 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:17:03.587663    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2dc993_51e2_4234_a738_9dd1d61891df.slice/crio-b4c95ead9f80b1914845f650baa10951d1de7e936aa3f60e84cd7444f1d13736 WatchSource:0}: Error finding container b4c95ead9f80b1914845f650baa10951d1de7e936aa3f60e84cd7444f1d13736: Status 404 returned error can't find the container with id b4c95ead9f80b1914845f650baa10951d1de7e936aa3f60e84cd7444f1d13736
Sep 19 15:17:04.018366 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:04.018325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2p7w" event={"ID":"bb2dc993-51e2-4234-a738-9dd1d61891df","Type":"ContainerStarted","Data":"b4c95ead9f80b1914845f650baa10951d1de7e936aa3f60e84cd7444f1d13736"}
Sep 19 15:17:06.026884 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:06.026850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2p7w" event={"ID":"bb2dc993-51e2-4234-a738-9dd1d61891df","Type":"ContainerStarted","Data":"c1d65e4b10336f1d31a0c39f69bd5b46dd31b9a0799227740e7fe55e7b9dec19"}
Sep 19 15:17:06.027306 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:06.026897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2p7w" event={"ID":"bb2dc993-51e2-4234-a738-9dd1d61891df","Type":"ContainerStarted","Data":"86f86efa6e08920587f2fded2d0713ca655cabf180c2080a3072b7d906f0f847"}
Sep 19 15:17:06.027306 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:06.026974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:17:06.044423 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:06.043889    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z2p7w" podStartSLOduration=251.6014643 podStartE2EDuration="4m13.043872518s" totalImagesPullingTime="1.442408218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:17:06.04337093 +0000 UTC m=+287.936934570" watchObservedRunningTime="2026-09-19 15:17:06.043872518 +0000 UTC m=+287.937436175"
Sep 19 15:17:09.315942 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.315904    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"]
Sep 19 15:17:09.321563 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.321543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.323562 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.323446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 19 15:17:09.323562 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.323451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 19 15:17:09.323562 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.323520    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 19 15:17:09.324100 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.324082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8jxk2\""
Sep 19 15:17:09.324195 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.324124    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 15:17:09.324195 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.324139    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 15:17:09.324296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.324268    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 15:17:09.326167 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.326145    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zsxd4"]
Sep 19 15:17:09.334265 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.334184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.334364 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.334287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"]
Sep 19 15:17:09.336213 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.336190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w8ll2\""
Sep 19 15:17:09.336439 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.336421    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 15:17:09.336587 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.336574    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 15:17:09.337255 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.337102    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 15:17:09.464174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84c09b31-d6a3-42a3-9de5-5184c4e3b893-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.464174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8321440-eed7-41cd-8dfa-68c507247399-metrics-client-ca\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464347 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-root\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464347 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-textfile\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464347 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464347 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngt87\" (UniqueName: \"kubernetes.io/projected/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-api-access-ngt87\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.464479 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-tls\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464479 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.464479 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.464479 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-wtmp\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464650 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464650 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-sys\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464650 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84c09b31-d6a3-42a3-9de5-5184c4e3b893-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.464650 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f25vg\" (UniqueName: \"kubernetes.io/projected/b8321440-eed7-41cd-8dfa-68c507247399-kube-api-access-f25vg\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.464650 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.464634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.565049 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-tls\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565198 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.565198 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.565198 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-wtmp\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565198 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565198 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-sys\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565198 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84c09b31-d6a3-42a3-9de5-5184c4e3b893-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.565538 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f25vg\" (UniqueName: \"kubernetes.io/projected/b8321440-eed7-41cd-8dfa-68c507247399-kube-api-access-f25vg\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565538 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.565538 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84c09b31-d6a3-42a3-9de5-5184c4e3b893-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.565538 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8321440-eed7-41cd-8dfa-68c507247399-metrics-client-ca\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565538 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-root\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565538 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-wtmp\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565538 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-textfile\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565884 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-textfile\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.565884 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-root\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.566004 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84c09b31-d6a3-42a3-9de5-5184c4e3b893-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.566004 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.566004 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.566004 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.565979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngt87\" (UniqueName: \"kubernetes.io/projected/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-api-access-ngt87\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.566175 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.566015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8321440-eed7-41cd-8dfa-68c507247399-sys\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.566350 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.566323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84c09b31-d6a3-42a3-9de5-5184c4e3b893-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.566680 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.566661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.566986 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.566964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8321440-eed7-41cd-8dfa-68c507247399-metrics-client-ca\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.567798 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.567768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-tls\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.568060 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.568042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.568274 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.568258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8321440-eed7-41cd-8dfa-68c507247399-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.568720 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.568701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.572492 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.572472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngt87\" (UniqueName: \"kubernetes.io/projected/84c09b31-d6a3-42a3-9de5-5184c4e3b893-kube-api-access-ngt87\") pod \"kube-state-metrics-84c885658c-cmvt4\" (UID: \"84c09b31-d6a3-42a3-9de5-5184c4e3b893\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.572560 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.572475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f25vg\" (UniqueName: \"kubernetes.io/projected/b8321440-eed7-41cd-8dfa-68c507247399-kube-api-access-f25vg\") pod \"node-exporter-zsxd4\" (UID: \"b8321440-eed7-41cd-8dfa-68c507247399\") " pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.638460 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.638443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"
Sep 19 15:17:09.646695 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.646681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zsxd4"
Sep 19 15:17:09.659541 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:17:09.659519    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8321440_eed7_41cd_8dfa_68c507247399.slice/crio-942832b5f891a1f2166da64aacb8a08416886efa24c1098bd23f480f426d93ef WatchSource:0}: Error finding container 942832b5f891a1f2166da64aacb8a08416886efa24c1098bd23f480f426d93ef: Status 404 returned error can't find the container with id 942832b5f891a1f2166da64aacb8a08416886efa24c1098bd23f480f426d93ef
Sep 19 15:17:09.913656 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:09.913554    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-cmvt4"]
Sep 19 15:17:09.916305 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:17:09.916280    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c09b31_d6a3_42a3_9de5_5184c4e3b893.slice/crio-d4705945af3701116065222c0457b46686f3e1d3a0925fb031137e007c85a6b9 WatchSource:0}: Error finding container d4705945af3701116065222c0457b46686f3e1d3a0925fb031137e007c85a6b9: Status 404 returned error can't find the container with id d4705945af3701116065222c0457b46686f3e1d3a0925fb031137e007c85a6b9
Sep 19 15:17:10.041393 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.041355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4" event={"ID":"84c09b31-d6a3-42a3-9de5-5184c4e3b893","Type":"ContainerStarted","Data":"d4705945af3701116065222c0457b46686f3e1d3a0925fb031137e007c85a6b9"}
Sep 19 15:17:10.042425 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.042404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsxd4" event={"ID":"b8321440-eed7-41cd-8dfa-68c507247399","Type":"ContainerStarted","Data":"942832b5f891a1f2166da64aacb8a08416886efa24c1098bd23f480f426d93ef"}
Sep 19 15:17:10.387805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.387772    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:17:10.391202 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.391176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.393275 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.393108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 19 15:17:10.393275 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.393160    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 19 15:17:10.393275 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.393177    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 19 15:17:10.393275 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.393183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 19 15:17:10.393565 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.393417    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 19 15:17:10.393565 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.393555    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-flcz5\""
Sep 19 15:17:10.393699 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.393588    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 19 15:17:10.393960 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.393928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 19 15:17:10.394663 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.394539    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 19 15:17:10.398098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.398071    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 19 15:17:10.407436 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.407415    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:17:10.472929 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.472888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.472981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-out\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-web-config\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-volume\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwfjd\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-kube-api-access-qwfjd\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473560 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.473560 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.473336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.513193 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.513154    2579 patch_prober.go:28] interesting pod/image-registry-64d48d969c-vl59s container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 19 15:17:10.513342 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.513211    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64d48d969c-vl59s" podUID="e3436924-04a9-449d-8276-4d71763093dc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 15:17:10.574206 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574358 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574358 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-out\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574358 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-web-config\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574358 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574358 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-volume\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574358 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwfjd\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-kube-api-access-qwfjd\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.574704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.574497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.575347 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.575196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.575347 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.575221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.575495 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.575347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.580278 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.580252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.581262 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.580840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.581262 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.580841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-out\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.581262 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.580867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.581457 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.581271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.581457 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.581335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.581457 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.581414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.582183 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.582162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-web-config\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.582924 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.582888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-volume\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.584978 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.584940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwfjd\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-kube-api-access-qwfjd\") pod \"alertmanager-main-0\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.701127 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.701095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:17:10.986584 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:10.986560    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:17:11.048245 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:11.048214    2579 generic.go:356] "Generic (PLEG): container finished" podID="b8321440-eed7-41cd-8dfa-68c507247399" containerID="94275e95d0a600779e4c4e022a50322092c25674a29d8a074a74865cc7f492d4" exitCode=0
Sep 19 15:17:11.048395 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:11.048272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsxd4" event={"ID":"b8321440-eed7-41cd-8dfa-68c507247399","Type":"ContainerDied","Data":"94275e95d0a600779e4c4e022a50322092c25674a29d8a074a74865cc7f492d4"}
Sep 19 15:17:11.153477 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:17:11.153440    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ca7ae9_7ce3_486a_840f_6ec5d4f26f95.slice/crio-ed9ec60ce871f59eec9c1b16148c86fa94087ccb0d0d984625eb222be83badb2 WatchSource:0}: Error finding container ed9ec60ce871f59eec9c1b16148c86fa94087ccb0d0d984625eb222be83badb2: Status 404 returned error can't find the container with id ed9ec60ce871f59eec9c1b16148c86fa94087ccb0d0d984625eb222be83badb2
Sep 19 15:17:11.957924 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:11.957896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64d48d969c-vl59s"
Sep 19 15:17:12.055174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:12.055135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsxd4" event={"ID":"b8321440-eed7-41cd-8dfa-68c507247399","Type":"ContainerStarted","Data":"2bcda1ad002b12fdad23dbd397cddebe25c2edf8f6666e04c02aab1b802504ee"}
Sep 19 15:17:12.055344 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:12.055208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsxd4" event={"ID":"b8321440-eed7-41cd-8dfa-68c507247399","Type":"ContainerStarted","Data":"fc9e965184baa61a6358176a571af7ef57659a9777b6c19ff51bdde5a784415e"}
Sep 19 15:17:12.057587 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:12.057558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4" event={"ID":"84c09b31-d6a3-42a3-9de5-5184c4e3b893","Type":"ContainerStarted","Data":"dfb0a57c43dd6025425b066546056aa847b22101c9b5af381e7e52dba01781f8"}
Sep 19 15:17:12.057710 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:12.057667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4" event={"ID":"84c09b31-d6a3-42a3-9de5-5184c4e3b893","Type":"ContainerStarted","Data":"00624e0ef074d794e3b97a17b46d8ed175ee3032f03c6848c64a501baf539288"}
Sep 19 15:17:12.057781 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:12.057709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4" event={"ID":"84c09b31-d6a3-42a3-9de5-5184c4e3b893","Type":"ContainerStarted","Data":"dc6f0aaa7244ecba6209e65e19614344892451b66736e34b8adc67faa6bab8b3"}
Sep 19 15:17:12.058924 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:12.058882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerStarted","Data":"ed9ec60ce871f59eec9c1b16148c86fa94087ccb0d0d984625eb222be83badb2"}
Sep 19 15:17:12.072446 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:12.072396    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zsxd4" podStartSLOduration=1.9951368729999999 podStartE2EDuration="3.072378477s" totalImagesPullingTime="1.002852824s" totalInitContainerRuntime="74.38878ms" isStatefulPod=true podCreationTimestamp="2026-09-19 15:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:17:12.070623715 +0000 UTC m=+293.964187356" watchObservedRunningTime="2026-09-19 15:17:12.072378477 +0000 UTC m=+293.965942118"
Sep 19 15:17:12.088962 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:12.088919    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-cmvt4" podStartSLOduration=1.799306118 podStartE2EDuration="3.088902421s" totalImagesPullingTime="1.289596303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:17:12.087478434 +0000 UTC m=+293.981042076" watchObservedRunningTime="2026-09-19 15:17:12.088902421 +0000 UTC m=+293.982466062"
Sep 19 15:17:13.063533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:13.063499    2579 generic.go:356] "Generic (PLEG): container finished" podID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerID="df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2" exitCode=0
Sep 19 15:17:13.063941 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:13.063584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerDied","Data":"df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2"}
Sep 19 15:17:15.074477 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:15.074452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerStarted","Data":"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311"}
Sep 19 15:17:15.074804 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:15.074512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerStarted","Data":"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d"}
Sep 19 15:17:15.074804 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:15.074537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerStarted","Data":"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe"}
Sep 19 15:17:15.074804 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:15.074558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerStarted","Data":"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f"}
Sep 19 15:17:15.074804 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:15.074582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerStarted","Data":"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619"}
Sep 19 15:17:16.033134 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:16.033110    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z2p7w"
Sep 19 15:17:16.083271 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:16.083245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerStarted","Data":"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d"}
Sep 19 15:17:16.107791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:16.107736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.303504142 podStartE2EDuration="6.107719935s" totalImagesPullingTime="3.785583605s" totalInitContainerRuntime="18.632188ms" isStatefulPod=false podCreationTimestamp="2026-09-19 15:17:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:17:16.106480846 +0000 UTC m=+298.000044481" watchObservedRunningTime="2026-09-19 15:17:16.107719935 +0000 UTC m=+298.001283573"
Sep 19 15:17:20.067622 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:20.067577    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:17:20.068014 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:20.067971    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:17:20.070532 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:20.070513    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 15:17:26.461138 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:26.461099    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" podUID="91c52b5e-06d8-4c9c-adfb-7da8f962f48c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 15:17:36.461231 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:36.461182    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" podUID="91c52b5e-06d8-4c9c-adfb-7da8f962f48c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 15:17:46.460889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:46.460849    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" podUID="91c52b5e-06d8-4c9c-adfb-7da8f962f48c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 19 15:17:46.461423 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:46.460943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr"
Sep 19 15:17:46.461564 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:46.461530    2579 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"db05992f18587665ee3c2d080b6a8cfd785e50df3a81d4ff5d1d197449393a89"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 19 15:17:46.461654 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:46.461617    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" podUID="91c52b5e-06d8-4c9c-adfb-7da8f962f48c" containerName="service-proxy" containerID="cri-o://db05992f18587665ee3c2d080b6a8cfd785e50df3a81d4ff5d1d197449393a89" gracePeriod=30
Sep 19 15:17:47.193931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:47.193900    2579 generic.go:356] "Generic (PLEG): container finished" podID="91c52b5e-06d8-4c9c-adfb-7da8f962f48c" containerID="db05992f18587665ee3c2d080b6a8cfd785e50df3a81d4ff5d1d197449393a89" exitCode=2
Sep 19 15:17:47.194098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:47.193950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" event={"ID":"91c52b5e-06d8-4c9c-adfb-7da8f962f48c","Type":"ContainerDied","Data":"db05992f18587665ee3c2d080b6a8cfd785e50df3a81d4ff5d1d197449393a89"}
Sep 19 15:17:47.194098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:17:47.193984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-767966c968-ghgfr" event={"ID":"91c52b5e-06d8-4c9c-adfb-7da8f962f48c","Type":"ContainerStarted","Data":"879b02fa05a239033928a5e9322565a5a782b7ec1bb991a5541f16aee8975e4a"}
Sep 19 15:18:29.527490 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:29.527456    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:18:29.528002 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:29.527755    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="alertmanager" containerID="cri-o://0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619" gracePeriod=120
Sep 19 15:18:29.528002 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:29.527848    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="prom-label-proxy" containerID="cri-o://6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d" gracePeriod=120
Sep 19 15:18:29.528002 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:29.527877    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy-web" containerID="cri-o://3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe" gracePeriod=120
Sep 19 15:18:29.528002 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:29.527842    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy-metric" containerID="cri-o://721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311" gracePeriod=120
Sep 19 15:18:29.528002 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:29.527902    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="config-reloader" containerID="cri-o://86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f" gracePeriod=120
Sep 19 15:18:29.528258 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:29.528011    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy" containerID="cri-o://846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d" gracePeriod=120
Sep 19 15:18:30.340451 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.340421    2579 generic.go:356] "Generic (PLEG): container finished" podID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerID="6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d" exitCode=0
Sep 19 15:18:30.340451 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.340443    2579 generic.go:356] "Generic (PLEG): container finished" podID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerID="846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d" exitCode=0
Sep 19 15:18:30.340451 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.340450    2579 generic.go:356] "Generic (PLEG): container finished" podID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerID="86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f" exitCode=0
Sep 19 15:18:30.340451 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.340456    2579 generic.go:356] "Generic (PLEG): container finished" podID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerID="0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619" exitCode=0
Sep 19 15:18:30.340737 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.340489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerDied","Data":"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d"}
Sep 19 15:18:30.340737 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.340547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerDied","Data":"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d"}
Sep 19 15:18:30.340737 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.340573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerDied","Data":"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f"}
Sep 19 15:18:30.340737 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.340596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerDied","Data":"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619"}
Sep 19 15:18:30.761596 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.761574    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:30.809473 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwfjd\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-kube-api-access-qwfjd\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809639 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-web\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809639 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-main-tls\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809639 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-main-db\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809803 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-volume\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809803 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-cluster-tls-config\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809803 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809713    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-tls-assets\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809803 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-metrics-client-ca\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809803 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-metric\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.809803 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.810084 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809829    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-web-config\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.810084 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-out\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.810084 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:18:30.810084 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.809904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-trusted-ca-bundle\") pod \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\" (UID: \"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95\") "
Sep 19 15:18:30.810436 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.810112    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-main-db\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.810489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.810419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:18:30.810641 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.810445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:18:30.813029 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.812975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:18:30.813029 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.813006    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-volume" (OuterVolumeSpecName: "config-volume") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:18:30.813378 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.813353    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:18:30.814001 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.813966    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:18:30.814086 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.814043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:18:30.814086 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.814066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:18:30.814362 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.814341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-kube-api-access-qwfjd" (OuterVolumeSpecName: "kube-api-access-qwfjd") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "kube-api-access-qwfjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:18:30.815006 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.814896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-out" (OuterVolumeSpecName: "config-out") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:18:30.818257 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.818231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:18:30.824471 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.824453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-web-config" (OuterVolumeSpecName: "web-config") pod "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" (UID: "26ca7ae9-7ce3-486a-840f-6ec5d4f26f95"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:18:30.911005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.910940    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-out\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.910963    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.910972    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwfjd\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-kube-api-access-qwfjd\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.910981    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.910989    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-main-tls\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.910997    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.911005    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-cluster-tls-config\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.911013    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-tls-assets\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.911021    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-metrics-client-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.911028    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.911038    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:30.911288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:30.911046    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95-web-config\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:18:31.347338 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.347303    2579 generic.go:356] "Generic (PLEG): container finished" podID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerID="721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311" exitCode=0
Sep 19 15:18:31.347338 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.347332    2579 generic.go:356] "Generic (PLEG): container finished" podID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerID="3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe" exitCode=0
Sep 19 15:18:31.347523 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.347411    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.347523 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.347403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerDied","Data":"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311"}
Sep 19 15:18:31.347635 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.347546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerDied","Data":"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe"}
Sep 19 15:18:31.347635 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.347572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26ca7ae9-7ce3-486a-840f-6ec5d4f26f95","Type":"ContainerDied","Data":"ed9ec60ce871f59eec9c1b16148c86fa94087ccb0d0d984625eb222be83badb2"}
Sep 19 15:18:31.347635 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.347624    2579 scope.go:117] "RemoveContainer" containerID="6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d"
Sep 19 15:18:31.355511 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.355496    2579 scope.go:117] "RemoveContainer" containerID="721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311"
Sep 19 15:18:31.361829 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.361807    2579 scope.go:117] "RemoveContainer" containerID="846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d"
Sep 19 15:18:31.368694 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.368664    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:18:31.369142 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.369129    2579 scope.go:117] "RemoveContainer" containerID="3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe"
Sep 19 15:18:31.372994 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.372974    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:18:31.375917 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.375904    2579 scope.go:117] "RemoveContainer" containerID="86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f"
Sep 19 15:18:31.382411 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.382394    2579 scope.go:117] "RemoveContainer" containerID="0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619"
Sep 19 15:18:31.388907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.388885    2579 scope.go:117] "RemoveContainer" containerID="df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2"
Sep 19 15:18:31.395783 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.395762    2579 scope.go:117] "RemoveContainer" containerID="6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d"
Sep 19 15:18:31.395992 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.395972    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:18:31.396129 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:18:31.396089    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d\": container with ID starting with 6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d not found: ID does not exist" containerID="6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d"
Sep 19 15:18:31.396232 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396140    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d"} err="failed to get container status \"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d\": rpc error: code = NotFound desc = could not find container \"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d\": container with ID starting with 6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d not found: ID does not exist"
Sep 19 15:18:31.396232 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396167    2579 scope.go:117] "RemoveContainer" containerID="721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311"
Sep 19 15:18:31.396616 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396576    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="init-config-reloader"
Sep 19 15:18:31.396698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396626    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="init-config-reloader"
Sep 19 15:18:31.396698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396644    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="config-reloader"
Sep 19 15:18:31.396698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396653    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="config-reloader"
Sep 19 15:18:31.396698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396676    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy-metric"
Sep 19 15:18:31.396698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396684    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy-metric"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396706    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="alertmanager"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396713    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="alertmanager"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396727    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396735    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396755    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy-web"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396762    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy-web"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396771    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="prom-label-proxy"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396780    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="prom-label-proxy"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396883    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy-web"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396896    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="alertmanager"
Sep 19 15:18:31.396909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396903    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy"
Sep 19 15:18:31.397337 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396917    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="config-reloader"
Sep 19 15:18:31.397337 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396925    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="kube-rbac-proxy-metric"
Sep 19 15:18:31.397337 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.396933    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" containerName="prom-label-proxy"
Sep 19 15:18:31.399933 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:18:31.396570    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311\": container with ID starting with 721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311 not found: ID does not exist" containerID="721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311"
Sep 19 15:18:31.399933 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.399281    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311"} err="failed to get container status \"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311\": rpc error: code = NotFound desc = could not find container \"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311\": container with ID starting with 721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311 not found: ID does not exist"
Sep 19 15:18:31.399933 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.399310    2579 scope.go:117] "RemoveContainer" containerID="846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d"
Sep 19 15:18:31.400181 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:18:31.400154    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d\": container with ID starting with 846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d not found: ID does not exist" containerID="846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d"
Sep 19 15:18:31.400516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.400492    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d"} err="failed to get container status \"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d\": rpc error: code = NotFound desc = could not find container \"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d\": container with ID starting with 846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d not found: ID does not exist"
Sep 19 15:18:31.400559 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.400519    2579 scope.go:117] "RemoveContainer" containerID="3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe"
Sep 19 15:18:31.400816 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:18:31.400794    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe\": container with ID starting with 3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe not found: ID does not exist" containerID="3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe"
Sep 19 15:18:31.400889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.400824    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe"} err="failed to get container status \"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe\": rpc error: code = NotFound desc = could not find container \"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe\": container with ID starting with 3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe not found: ID does not exist"
Sep 19 15:18:31.400889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.400841    2579 scope.go:117] "RemoveContainer" containerID="86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f"
Sep 19 15:18:31.401101 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:18:31.401084    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f\": container with ID starting with 86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f not found: ID does not exist" containerID="86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f"
Sep 19 15:18:31.401142 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.401107    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f"} err="failed to get container status \"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f\": rpc error: code = NotFound desc = could not find container \"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f\": container with ID starting with 86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f not found: ID does not exist"
Sep 19 15:18:31.401142 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.401121    2579 scope.go:117] "RemoveContainer" containerID="0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619"
Sep 19 15:18:31.401364 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:18:31.401346    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619\": container with ID starting with 0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619 not found: ID does not exist" containerID="0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619"
Sep 19 15:18:31.401412 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.401370    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619"} err="failed to get container status \"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619\": rpc error: code = NotFound desc = could not find container \"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619\": container with ID starting with 0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619 not found: ID does not exist"
Sep 19 15:18:31.401412 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.401386    2579 scope.go:117] "RemoveContainer" containerID="df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2"
Sep 19 15:18:31.401664 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:18:31.401645    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2\": container with ID starting with df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2 not found: ID does not exist" containerID="df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2"
Sep 19 15:18:31.401743 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.401667    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2"} err="failed to get container status \"df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2\": rpc error: code = NotFound desc = could not find container \"df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2\": container with ID starting with df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2 not found: ID does not exist"
Sep 19 15:18:31.401743 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.401681    2579 scope.go:117] "RemoveContainer" containerID="6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d"
Sep 19 15:18:31.401962 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.401936    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d"} err="failed to get container status \"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d\": rpc error: code = NotFound desc = could not find container \"6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d\": container with ID starting with 6d9b3d1ccfdbaa52916280820e7c47c6e2755088eec010961f089eb3338a079d not found: ID does not exist"
Sep 19 15:18:31.402006 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.401968    2579 scope.go:117] "RemoveContainer" containerID="721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311"
Sep 19 15:18:31.402241 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402225    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311"} err="failed to get container status \"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311\": rpc error: code = NotFound desc = could not find container \"721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311\": container with ID starting with 721bad24214ac0b4d4a7865d70c16cd2cdab83f497927b5e5b883dbe2f495311 not found: ID does not exist"
Sep 19 15:18:31.402286 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402242    2579 scope.go:117] "RemoveContainer" containerID="846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d"
Sep 19 15:18:31.402457 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402439    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d"} err="failed to get container status \"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d\": rpc error: code = NotFound desc = could not find container \"846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d\": container with ID starting with 846f67f7b47ac972d1d4d4bcc46062278dd1a90003d51a5bfe96042eda7c059d not found: ID does not exist"
Sep 19 15:18:31.402494 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402459    2579 scope.go:117] "RemoveContainer" containerID="3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe"
Sep 19 15:18:31.402714 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402695    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe"} err="failed to get container status \"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe\": rpc error: code = NotFound desc = could not find container \"3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe\": container with ID starting with 3ecf06f870d4b88f60cd5bad8debf43491a85e2f1ba146f93e9a10464c8a72fe not found: ID does not exist"
Sep 19 15:18:31.402761 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402714    2579 scope.go:117] "RemoveContainer" containerID="86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f"
Sep 19 15:18:31.402836 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.402918 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402901    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f"} err="failed to get container status \"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f\": rpc error: code = NotFound desc = could not find container \"86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f\": container with ID starting with 86e30d25281344f5ce4222ff6084836568856b0a9ad0c4391aca341aebfeb93f not found: ID does not exist"
Sep 19 15:18:31.402963 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.402920    2579 scope.go:117] "RemoveContainer" containerID="0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619"
Sep 19 15:18:31.403201 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.403177    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619"} err="failed to get container status \"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619\": rpc error: code = NotFound desc = could not find container \"0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619\": container with ID starting with 0e7c1a2d9be4872d0d1a23cf1f3c726c11772e923f752f5f6e057980b9e57619 not found: ID does not exist"
Sep 19 15:18:31.403280 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.403201    2579 scope.go:117] "RemoveContainer" containerID="df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2"
Sep 19 15:18:31.403470 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.403446    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2"} err="failed to get container status \"df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2\": rpc error: code = NotFound desc = could not find container \"df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2\": container with ID starting with df2ac639fee5d719c54deae0722290ee6b81dc7b01bddaf527d8e2dfd2e214c2 not found: ID does not exist"
Sep 19 15:18:31.404800 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.404784    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 19 15:18:31.404907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.404885    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 19 15:18:31.405004 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.404980    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 19 15:18:31.405070 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.405053    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 19 15:18:31.405169 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.405154    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 19 15:18:31.405258 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.405237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 19 15:18:31.405338 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.405324    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-flcz5\""
Sep 19 15:18:31.405390 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.405371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 19 15:18:31.405517 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.405500    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 19 15:18:31.409910 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.409893    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 19 15:18:31.411252 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.411231    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:18:31.515227 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-web-config\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da849908-016d-4220-b2f2-f1dc5f2dc36e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da849908-016d-4220-b2f2-f1dc5f2dc36e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515554 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da849908-016d-4220-b2f2-f1dc5f2dc36e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515554 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da849908-016d-4220-b2f2-f1dc5f2dc36e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515554 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-config-volume\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515554 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da849908-016d-4220-b2f2-f1dc5f2dc36e-config-out\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515554 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rthj2\" (UniqueName: \"kubernetes.io/projected/da849908-016d-4220-b2f2-f1dc5f2dc36e-kube-api-access-rthj2\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.515722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.515554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616467 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616467 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da849908-016d-4220-b2f2-f1dc5f2dc36e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616467 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da849908-016d-4220-b2f2-f1dc5f2dc36e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616467 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-config-volume\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616696 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da849908-016d-4220-b2f2-f1dc5f2dc36e-config-out\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616696 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rthj2\" (UniqueName: \"kubernetes.io/projected/da849908-016d-4220-b2f2-f1dc5f2dc36e-kube-api-access-rthj2\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616696 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616696 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-web-config\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616878 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616878 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da849908-016d-4220-b2f2-f1dc5f2dc36e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.616878 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.616795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da849908-016d-4220-b2f2-f1dc5f2dc36e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.617728 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.617703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da849908-016d-4220-b2f2-f1dc5f2dc36e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.617834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.617809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da849908-016d-4220-b2f2-f1dc5f2dc36e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.619566 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.619540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da849908-016d-4220-b2f2-f1dc5f2dc36e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.619566 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.619560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.619737 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.619633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-config-volume\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.619853 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.619828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.619940 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.619915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da849908-016d-4220-b2f2-f1dc5f2dc36e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.620110 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.620083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da849908-016d-4220-b2f2-f1dc5f2dc36e-config-out\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.620201 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.620117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.620480 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.620462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.620646 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.620626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-web-config\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.621769 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.621739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da849908-016d-4220-b2f2-f1dc5f2dc36e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.625497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.625479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rthj2\" (UniqueName: \"kubernetes.io/projected/da849908-016d-4220-b2f2-f1dc5f2dc36e-kube-api-access-rthj2\") pod \"alertmanager-main-0\" (UID: \"da849908-016d-4220-b2f2-f1dc5f2dc36e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.713583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.713561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 15:18:31.987995 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:31.987938    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 15:18:31.990173 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:18:31.990146    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda849908_016d_4220_b2f2_f1dc5f2dc36e.slice/crio-d779f417825aec45dbc614f7cd1b7c49d761624afcd560d037d093880157d270 WatchSource:0}: Error finding container d779f417825aec45dbc614f7cd1b7c49d761624afcd560d037d093880157d270: Status 404 returned error can't find the container with id d779f417825aec45dbc614f7cd1b7c49d761624afcd560d037d093880157d270
Sep 19 15:18:32.169092 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:32.169060    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ca7ae9-7ce3-486a-840f-6ec5d4f26f95" path="/var/lib/kubelet/pods/26ca7ae9-7ce3-486a-840f-6ec5d4f26f95/volumes"
Sep 19 15:18:32.353081 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:32.352997    2579 generic.go:356] "Generic (PLEG): container finished" podID="da849908-016d-4220-b2f2-f1dc5f2dc36e" containerID="6b51e41daba15e296d58177983515406c2c3944dec07ce8e8044d8686f9d71c3" exitCode=0
Sep 19 15:18:32.353081 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:32.353068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da849908-016d-4220-b2f2-f1dc5f2dc36e","Type":"ContainerDied","Data":"6b51e41daba15e296d58177983515406c2c3944dec07ce8e8044d8686f9d71c3"}
Sep 19 15:18:32.353261 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:32.353139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da849908-016d-4220-b2f2-f1dc5f2dc36e","Type":"ContainerStarted","Data":"d779f417825aec45dbc614f7cd1b7c49d761624afcd560d037d093880157d270"}
Sep 19 15:18:33.360100 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.360070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da849908-016d-4220-b2f2-f1dc5f2dc36e","Type":"ContainerStarted","Data":"18b9fa4a38fd370462c17a3eb248d9c088f125d9a877b7d86623f0c270b7a24c"}
Sep 19 15:18:33.360456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.360131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da849908-016d-4220-b2f2-f1dc5f2dc36e","Type":"ContainerStarted","Data":"29079f0469aeabcc10e0082393ccd3f764f2396a381b258fd6547b71bb647ae3"}
Sep 19 15:18:33.360456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.360156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da849908-016d-4220-b2f2-f1dc5f2dc36e","Type":"ContainerStarted","Data":"0407bdfa7d103f4ba05f151a4d0d26ad5eab989d7462416f31a2ec6953a08411"}
Sep 19 15:18:33.360456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.360180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da849908-016d-4220-b2f2-f1dc5f2dc36e","Type":"ContainerStarted","Data":"591daebd7adaf0d4986d2cb4dfaeccaca0d734e975bec4d03cac28c257c1b07c"}
Sep 19 15:18:33.360456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.360209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da849908-016d-4220-b2f2-f1dc5f2dc36e","Type":"ContainerStarted","Data":"a9194d707ad490049f945c05317068cce18e21f16a88acb4ba87c6e6ed0c5fbe"}
Sep 19 15:18:33.360456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.360232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da849908-016d-4220-b2f2-f1dc5f2dc36e","Type":"ContainerStarted","Data":"f13e62535821f08a38b3e6e9f1fb7948f20cab679eed52cc45b25d708afef790"}
Sep 19 15:18:33.388853 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.388805    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.373445121 podStartE2EDuration="2.388789753s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.344632ms" isStatefulPod=false podCreationTimestamp="2026-09-19 15:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:18:33.38199169 +0000 UTC m=+375.275555328" watchObservedRunningTime="2026-09-19 15:18:33.388789753 +0000 UTC m=+375.282353393"
Sep 19 15:18:33.547479 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.547445    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-784764ddc8-5m2rp"]
Sep 19 15:18:33.550654 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.550634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.552780 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.552760    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 19 15:18:33.552912 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.552829    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gck7n\""
Sep 19 15:18:33.552912 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.552866    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 19 15:18:33.553014 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.553000    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 19 15:18:33.553073 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.553059    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 19 15:18:33.553119 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.553068    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 19 15:18:33.557109 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.557092    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 19 15:18:33.574362 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.574321    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-784764ddc8-5m2rp"]
Sep 19 15:18:33.631236 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.631182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-federate-client-tls\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.631236 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.631209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpg5v\" (UniqueName: \"kubernetes.io/projected/0a124547-3cfa-498b-8230-f3edd90c5751-kube-api-access-lpg5v\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.631236 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.631229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-serving-certs-ca-bundle\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.631389 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.631304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-telemeter-client-tls\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.631389 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.631328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-secret-telemeter-client\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.631389 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.631345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-metrics-client-ca\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.631488 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.631390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.631488 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.631409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-telemeter-trusted-ca-bundle\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.731945 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.731925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-telemeter-client-tls\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.731955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-secret-telemeter-client\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.731974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-metrics-client-ca\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.731994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.732021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-telemeter-trusted-ca-bundle\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.732049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-federate-client-tls\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732258 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.732077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpg5v\" (UniqueName: \"kubernetes.io/projected/0a124547-3cfa-498b-8230-f3edd90c5751-kube-api-access-lpg5v\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732258 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.732107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-serving-certs-ca-bundle\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732904 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.732880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-serving-certs-ca-bundle\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.732904 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.732893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-metrics-client-ca\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.733128 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.733108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a124547-3cfa-498b-8230-f3edd90c5751-telemeter-trusted-ca-bundle\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.734632 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.734592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.734698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.734655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-telemeter-client-tls\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.735086 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.735065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-federate-client-tls\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.735227 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.735207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0a124547-3cfa-498b-8230-f3edd90c5751-secret-telemeter-client\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.739238 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.739217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpg5v\" (UniqueName: \"kubernetes.io/projected/0a124547-3cfa-498b-8230-f3edd90c5751-kube-api-access-lpg5v\") pod \"telemeter-client-784764ddc8-5m2rp\" (UID: \"0a124547-3cfa-498b-8230-f3edd90c5751\") " pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:33.860422 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:33.860393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp"
Sep 19 15:18:34.134133 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:34.134108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-784764ddc8-5m2rp"]
Sep 19 15:18:34.136400 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:18:34.136374    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a124547_3cfa_498b_8230_f3edd90c5751.slice/crio-53b3f8681867b4c779c1fb6e4699f41af31d386cab6f5b1879d2c3b4a6b3bc4c WatchSource:0}: Error finding container 53b3f8681867b4c779c1fb6e4699f41af31d386cab6f5b1879d2c3b4a6b3bc4c: Status 404 returned error can't find the container with id 53b3f8681867b4c779c1fb6e4699f41af31d386cab6f5b1879d2c3b4a6b3bc4c
Sep 19 15:18:34.138334 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:34.138319    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 15:18:34.364674 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:34.364643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp" event={"ID":"0a124547-3cfa-498b-8230-f3edd90c5751","Type":"ContainerStarted","Data":"53b3f8681867b4c779c1fb6e4699f41af31d386cab6f5b1879d2c3b4a6b3bc4c"}
Sep 19 15:18:36.374687 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:36.374588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp" event={"ID":"0a124547-3cfa-498b-8230-f3edd90c5751","Type":"ContainerStarted","Data":"3b5bd3be82cc336749566d0dd5e32e1287d2edc090fa8545324032782ddf2c5e"}
Sep 19 15:18:36.374687 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:36.374665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp" event={"ID":"0a124547-3cfa-498b-8230-f3edd90c5751","Type":"ContainerStarted","Data":"79b69ec5eff5f35943ed35fdf5e9a0d7851cf7cfe2f40cab3a6e031abfc04509"}
Sep 19 15:18:36.375045 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:36.374694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp" event={"ID":"0a124547-3cfa-498b-8230-f3edd90c5751","Type":"ContainerStarted","Data":"fc2aa7a40a90552cc22f765703eddeb41872c9a58766540577d38c811f051971"}
Sep 19 15:18:36.394215 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:18:36.394159    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-784764ddc8-5m2rp" podStartSLOduration=1.5996468670000001 podStartE2EDuration="3.394145897s" totalImagesPullingTime="1.79449903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:18:36.392120679 +0000 UTC m=+378.285684319" watchObservedRunningTime="2026-09-19 15:18:36.394145897 +0000 UTC m=+378.287709548"
Sep 19 15:21:17.417473 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.417411    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q7hqv"]
Sep 19 15:21:17.420435 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.420418    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-q7hqv"
Sep 19 15:21:17.422407 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.422390    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 19 15:21:17.422407 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.422401    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 19 15:21:17.422928 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.422911    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7z4p6\""
Sep 19 15:21:17.426917 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.426889    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q7hqv"]
Sep 19 15:21:17.554712 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.554684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jflsv\" (UniqueName: \"kubernetes.io/projected/61e4b631-81be-4643-94a9-ddaa0220038c-kube-api-access-jflsv\") pod \"cert-manager-55b4d954d5-q7hqv\" (UID: \"61e4b631-81be-4643-94a9-ddaa0220038c\") " pod="cert-manager/cert-manager-55b4d954d5-q7hqv"
Sep 19 15:21:17.554821 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.554717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61e4b631-81be-4643-94a9-ddaa0220038c-bound-sa-token\") pod \"cert-manager-55b4d954d5-q7hqv\" (UID: \"61e4b631-81be-4643-94a9-ddaa0220038c\") " pod="cert-manager/cert-manager-55b4d954d5-q7hqv"
Sep 19 15:21:17.655586 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.655558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jflsv\" (UniqueName: \"kubernetes.io/projected/61e4b631-81be-4643-94a9-ddaa0220038c-kube-api-access-jflsv\") pod \"cert-manager-55b4d954d5-q7hqv\" (UID: \"61e4b631-81be-4643-94a9-ddaa0220038c\") " pod="cert-manager/cert-manager-55b4d954d5-q7hqv"
Sep 19 15:21:17.655718 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.655591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61e4b631-81be-4643-94a9-ddaa0220038c-bound-sa-token\") pod \"cert-manager-55b4d954d5-q7hqv\" (UID: \"61e4b631-81be-4643-94a9-ddaa0220038c\") " pod="cert-manager/cert-manager-55b4d954d5-q7hqv"
Sep 19 15:21:17.662853 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.662825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61e4b631-81be-4643-94a9-ddaa0220038c-bound-sa-token\") pod \"cert-manager-55b4d954d5-q7hqv\" (UID: \"61e4b631-81be-4643-94a9-ddaa0220038c\") " pod="cert-manager/cert-manager-55b4d954d5-q7hqv"
Sep 19 15:21:17.662935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.662922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jflsv\" (UniqueName: \"kubernetes.io/projected/61e4b631-81be-4643-94a9-ddaa0220038c-kube-api-access-jflsv\") pod \"cert-manager-55b4d954d5-q7hqv\" (UID: \"61e4b631-81be-4643-94a9-ddaa0220038c\") " pod="cert-manager/cert-manager-55b4d954d5-q7hqv"
Sep 19 15:21:17.729813 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.729792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-q7hqv"
Sep 19 15:21:17.995232 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:17.995173    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q7hqv"]
Sep 19 15:21:17.997352 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:21:17.997328    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61e4b631_81be_4643_94a9_ddaa0220038c.slice/crio-1d51fdd8bbb2330b9277855a391881ccb50ed500fa430e005d363920fe72ee0c WatchSource:0}: Error finding container 1d51fdd8bbb2330b9277855a391881ccb50ed500fa430e005d363920fe72ee0c: Status 404 returned error can't find the container with id 1d51fdd8bbb2330b9277855a391881ccb50ed500fa430e005d363920fe72ee0c
Sep 19 15:21:18.939925 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:18.939881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-q7hqv" event={"ID":"61e4b631-81be-4643-94a9-ddaa0220038c","Type":"ContainerStarted","Data":"1d51fdd8bbb2330b9277855a391881ccb50ed500fa430e005d363920fe72ee0c"}
Sep 19 15:21:20.948893 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:20.948858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-q7hqv" event={"ID":"61e4b631-81be-4643-94a9-ddaa0220038c","Type":"ContainerStarted","Data":"7c36914188c4e1a0412042314b40ca819589f7f81127a82eccdf3be852a93c6a"}
Sep 19 15:21:20.962660 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:20.962617    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-q7hqv" podStartSLOduration=1.202749991 podStartE2EDuration="3.962585697s" totalImagesPullingTime="2.759835706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:21:20.961671676 +0000 UTC m=+542.855235322" watchObservedRunningTime="2026-09-19 15:21:20.962585697 +0000 UTC m=+542.856149336"
Sep 19 15:21:21.380350 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.380323    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"]
Sep 19 15:21:21.383337 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.383315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"
Sep 19 15:21:21.385178 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.385137    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 19 15:21:21.385320 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.385302    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9nbgr\""
Sep 19 15:21:21.385423 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.385400    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 19 15:21:21.391506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.391486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"]
Sep 19 15:21:21.486336 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.486304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/125d5de7-7a89-414e-af80-65298e46035d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5wc8t\" (UID: \"125d5de7-7a89-414e-af80-65298e46035d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"
Sep 19 15:21:21.486462 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.486348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9lm\" (UniqueName: \"kubernetes.io/projected/125d5de7-7a89-414e-af80-65298e46035d-kube-api-access-7t9lm\") pod \"openshift-lws-operator-fd8ccff4c-5wc8t\" (UID: \"125d5de7-7a89-414e-af80-65298e46035d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"
Sep 19 15:21:21.587314 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.587292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/125d5de7-7a89-414e-af80-65298e46035d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5wc8t\" (UID: \"125d5de7-7a89-414e-af80-65298e46035d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"
Sep 19 15:21:21.587398 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.587324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9lm\" (UniqueName: \"kubernetes.io/projected/125d5de7-7a89-414e-af80-65298e46035d-kube-api-access-7t9lm\") pod \"openshift-lws-operator-fd8ccff4c-5wc8t\" (UID: \"125d5de7-7a89-414e-af80-65298e46035d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"
Sep 19 15:21:21.587637 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.587621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/125d5de7-7a89-414e-af80-65298e46035d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5wc8t\" (UID: \"125d5de7-7a89-414e-af80-65298e46035d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"
Sep 19 15:21:21.594366 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.594337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9lm\" (UniqueName: \"kubernetes.io/projected/125d5de7-7a89-414e-af80-65298e46035d-kube-api-access-7t9lm\") pod \"openshift-lws-operator-fd8ccff4c-5wc8t\" (UID: \"125d5de7-7a89-414e-af80-65298e46035d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"
Sep 19 15:21:21.691751 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.691732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"
Sep 19 15:21:21.958037 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:21.957970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t"]
Sep 19 15:21:21.960897 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:21:21.960873    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod125d5de7_7a89_414e_af80_65298e46035d.slice/crio-512e76fe70f20307648a9b6cdb4084f14882db4cc68a4d39dc29aa4e4e94043c WatchSource:0}: Error finding container 512e76fe70f20307648a9b6cdb4084f14882db4cc68a4d39dc29aa4e4e94043c: Status 404 returned error can't find the container with id 512e76fe70f20307648a9b6cdb4084f14882db4cc68a4d39dc29aa4e4e94043c
Sep 19 15:21:22.958624 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:22.958575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t" event={"ID":"125d5de7-7a89-414e-af80-65298e46035d","Type":"ContainerStarted","Data":"512e76fe70f20307648a9b6cdb4084f14882db4cc68a4d39dc29aa4e4e94043c"}
Sep 19 15:21:24.967938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:24.967900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t" event={"ID":"125d5de7-7a89-414e-af80-65298e46035d","Type":"ContainerStarted","Data":"f54003d4e0c1c85c1422663dea1f42bd27a8775e0de1b5e72262ce642faeb612"}
Sep 19 15:21:24.981781 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:24.981708    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5wc8t" podStartSLOduration=1.612325402 podStartE2EDuration="3.981693207s" totalImagesPullingTime="2.369367805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:21:24.981432544 +0000 UTC m=+546.874996194" watchObservedRunningTime="2026-09-19 15:21:24.981693207 +0000 UTC m=+546.875256846"
Sep 19 15:21:44.209675 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.209646    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"]
Sep 19 15:21:44.212979 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.212961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.216125 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.216064    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 19 15:21:44.218492 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.218468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 19 15:21:44.218596 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.218488    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 19 15:21:44.218596 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.218490    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 19 15:21:44.218596 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.218574    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6pmc7\""
Sep 19 15:21:44.225652 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.225631    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"]
Sep 19 15:21:44.244539 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.244517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-tmp\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.244667 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.244549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.244667 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.244571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.244667 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.244619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sfwb\" (UniqueName: \"kubernetes.io/projected/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-kube-api-access-5sfwb\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.244813 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.244714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-manifests\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.346700 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.346661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.347136 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.346758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.347264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.347193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sfwb\" (UniqueName: \"kubernetes.io/projected/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-kube-api-access-5sfwb\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.347264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.347245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-manifests\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.347366 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.347336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-tmp\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.347646 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.347581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-manifests\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.350316 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.350296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-tmp\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.350542 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.350521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.351034 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.351014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.356100 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.356082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sfwb\" (UniqueName: \"kubernetes.io/projected/2eb599ba-05b4-47de-a16a-9fcb1a3a2b37-kube-api-access-5sfwb\") pod \"opendatahub-operator-controller-manager-79d8c7f66c-hq2hz\" (UID: \"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37\") " pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.522425 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.522350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:44.795423 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:44.795400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"]
Sep 19 15:21:44.797732 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:21:44.797699    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb599ba_05b4_47de_a16a_9fcb1a3a2b37.slice/crio-5f31f599b3cdd1d6ec7cfa7a4ba42f90d376325daf268af19244146f366ed899 WatchSource:0}: Error finding container 5f31f599b3cdd1d6ec7cfa7a4ba42f90d376325daf268af19244146f366ed899: Status 404 returned error can't find the container with id 5f31f599b3cdd1d6ec7cfa7a4ba42f90d376325daf268af19244146f366ed899
Sep 19 15:21:45.039557 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:45.039526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz" event={"ID":"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37","Type":"ContainerStarted","Data":"5f31f599b3cdd1d6ec7cfa7a4ba42f90d376325daf268af19244146f366ed899"}
Sep 19 15:21:48.054948 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:48.054914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz" event={"ID":"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37","Type":"ContainerStarted","Data":"b6f168c12dc205f87a19b32599da65849e0a9d3dba5dd41f13db90d2ec463bae"}
Sep 19 15:21:49.060712 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:49.060674    2579 generic.go:356] "Generic (PLEG): container finished" podID="2eb599ba-05b4-47de-a16a-9fcb1a3a2b37" containerID="b6f168c12dc205f87a19b32599da65849e0a9d3dba5dd41f13db90d2ec463bae" exitCode=0
Sep 19 15:21:49.061061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:49.060755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz" event={"ID":"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37","Type":"ContainerDied","Data":"b6f168c12dc205f87a19b32599da65849e0a9d3dba5dd41f13db90d2ec463bae"}
Sep 19 15:21:50.067150 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:50.067116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz" event={"ID":"2eb599ba-05b4-47de-a16a-9fcb1a3a2b37","Type":"ContainerStarted","Data":"e8b78c4ac01cd2b2b8bcf71c1d4a46faaae40464c61cc619d27932db5a27f77c"}
Sep 19 15:21:50.067560 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:50.067240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:21:50.086910 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:21:50.086864    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz" podStartSLOduration=1.9511741169999999 podStartE2EDuration="6.086849619s" totalImagesPullingTime="3.206908438s" totalInitContainerRuntime="928.767064ms" isStatefulPod=false podCreationTimestamp="2026-09-19 15:21:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:21:50.085014005 +0000 UTC m=+571.978577647" watchObservedRunningTime="2026-09-19 15:21:50.086849619 +0000 UTC m=+571.980413258"
Sep 19 15:22:01.073542 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:01.073512    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79d8c7f66c-hq2hz"
Sep 19 15:22:04.706532 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.706496    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"]
Sep 19 15:22:04.709491 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.709471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.711735 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.711712    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 19 15:22:04.711929 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.711916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r5bbd\""
Sep 19 15:22:04.712500 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.712484    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 19 15:22:04.712553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.712486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 19 15:22:04.712616 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.712554    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 19 15:22:04.733553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.733532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"]
Sep 19 15:22:04.803185 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.803154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8393d6f3-9666-4eae-9964-af24387a39fb-tls-certs\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.803333 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.803232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5nh\" (UniqueName: \"kubernetes.io/projected/8393d6f3-9666-4eae-9964-af24387a39fb-kube-api-access-7w5nh\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.803333 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.803277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8393d6f3-9666-4eae-9964-af24387a39fb-tmp\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.904431 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.904401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5nh\" (UniqueName: \"kubernetes.io/projected/8393d6f3-9666-4eae-9964-af24387a39fb-kube-api-access-7w5nh\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.904549 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.904437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8393d6f3-9666-4eae-9964-af24387a39fb-tmp\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.904549 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.904465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8393d6f3-9666-4eae-9964-af24387a39fb-tls-certs\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.906825 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.906804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8393d6f3-9666-4eae-9964-af24387a39fb-tmp\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.906907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.906874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8393d6f3-9666-4eae-9964-af24387a39fb-tls-certs\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:04.919925 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:04.919896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5nh\" (UniqueName: \"kubernetes.io/projected/8393d6f3-9666-4eae-9964-af24387a39fb-kube-api-access-7w5nh\") pod \"kube-auth-proxy-dc4565f8d-dnjjq\" (UID: \"8393d6f3-9666-4eae-9964-af24387a39fb\") " pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:05.021167 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:05.021116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"
Sep 19 15:22:05.296624 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:05.296582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq"]
Sep 19 15:22:05.298859 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:22:05.298833    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8393d6f3_9666_4eae_9964_af24387a39fb.slice/crio-4b33a8d90bafac2c7a5c94569e72c70bd15157805a6b88104f9b7d7e84f33813 WatchSource:0}: Error finding container 4b33a8d90bafac2c7a5c94569e72c70bd15157805a6b88104f9b7d7e84f33813: Status 404 returned error can't find the container with id 4b33a8d90bafac2c7a5c94569e72c70bd15157805a6b88104f9b7d7e84f33813
Sep 19 15:22:06.136166 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:06.136093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq" event={"ID":"8393d6f3-9666-4eae-9964-af24387a39fb","Type":"ContainerStarted","Data":"4b33a8d90bafac2c7a5c94569e72c70bd15157805a6b88104f9b7d7e84f33813"}
Sep 19 15:22:08.146516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:08.146482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq" event={"ID":"8393d6f3-9666-4eae-9964-af24387a39fb","Type":"ContainerStarted","Data":"427a3d061af37c3aa99c86e2b1bf53051dea3afed85399271e51c6cb26f68ee0"}
Sep 19 15:22:08.162006 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:08.161963    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-dc4565f8d-dnjjq" podStartSLOduration=1.498271775 podStartE2EDuration="4.16195012s" totalImagesPullingTime="2.663678345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:22:08.159963378 +0000 UTC m=+590.053527017" watchObservedRunningTime="2026-09-19 15:22:08.16195012 +0000 UTC m=+590.055513759"
Sep 19 15:22:12.868345 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.868309    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"]
Sep 19 15:22:12.876416 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.876391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:12.879039 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.879012    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 19 15:22:12.879400 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.879382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 19 15:22:12.879517 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.879492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 19 15:22:12.879658 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.879637    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2fcld\""
Sep 19 15:22:12.880241 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.880221    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"]
Sep 19 15:22:12.967800 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.967774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-metrics-cert\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:12.967921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.967803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-manager-config\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:12.967921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.967817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-cert\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:12.967921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:12.967884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpcpk\" (UniqueName: \"kubernetes.io/projected/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-kube-api-access-kpcpk\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.068707 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.068680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-metrics-cert\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.068797 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.068710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-manager-config\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.068797 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.068726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-cert\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.068797 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.068762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpcpk\" (UniqueName: \"kubernetes.io/projected/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-kube-api-access-kpcpk\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.069381 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.069360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-manager-config\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.071271 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.071254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-metrics-cert\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.071363 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.071321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-cert\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.075836 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.075816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpcpk\" (UniqueName: \"kubernetes.io/projected/1d1484f1-0345-4655-a9e9-3117ce5f5e6c-kube-api-access-kpcpk\") pod \"lws-controller-manager-7b79856b6-l7kfn\" (UID: \"1d1484f1-0345-4655-a9e9-3117ce5f5e6c\") " pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.187446 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.187424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:13.454062 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:13.453951    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"]
Sep 19 15:22:13.456617 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:22:13.456559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1484f1_0345_4655_a9e9_3117ce5f5e6c.slice/crio-dc73fdd66b02cc278b4f5592b1572f0ccb2adce03cb85734e6df410aa02aebc2 WatchSource:0}: Error finding container dc73fdd66b02cc278b4f5592b1572f0ccb2adce03cb85734e6df410aa02aebc2: Status 404 returned error can't find the container with id dc73fdd66b02cc278b4f5592b1572f0ccb2adce03cb85734e6df410aa02aebc2
Sep 19 15:22:14.172121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:14.172090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn" event={"ID":"1d1484f1-0345-4655-a9e9-3117ce5f5e6c","Type":"ContainerStarted","Data":"dc73fdd66b02cc278b4f5592b1572f0ccb2adce03cb85734e6df410aa02aebc2"}
Sep 19 15:22:16.184129 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:16.184089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn" event={"ID":"1d1484f1-0345-4655-a9e9-3117ce5f5e6c","Type":"ContainerStarted","Data":"f5d6b964ec16a93f4eecf83f27d0ccf61fb209e3ee111052ee5424f9dcd9ba0b"}
Sep 19 15:22:16.184506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:16.184221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:22:16.199178 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:16.199134    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn" podStartSLOduration=2.394706116 podStartE2EDuration="4.199120955s" totalImagesPullingTime="1.804414839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:22:16.197505082 +0000 UTC m=+598.091068723" watchObservedRunningTime="2026-09-19 15:22:16.199120955 +0000 UTC m=+598.092684594"
Sep 19 15:22:20.098884 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:20.098856    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:22:20.099330 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:20.099180    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:22:27.194183 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:22:27.194147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b79856b6-l7kfn"
Sep 19 15:23:43.163296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.163264    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"]
Sep 19 15:23:43.166368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.166343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.168361 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.168342    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 19 15:23:43.168583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.168568    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 19 15:23:43.169150 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.169133    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x2x89\""
Sep 19 15:23:43.169221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.169152    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 19 15:23:43.169221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.169153    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 19 15:23:43.175704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.175683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"]
Sep 19 15:23:43.314790 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.314759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj55b\" (UniqueName: \"kubernetes.io/projected/02e89a48-96ff-4fde-8ff3-9d278994582e-kube-api-access-qj55b\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.315209 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.315170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/02e89a48-96ff-4fde-8ff3-9d278994582e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.315331 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.315223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/02e89a48-96ff-4fde-8ff3-9d278994582e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.415770 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.415705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/02e89a48-96ff-4fde-8ff3-9d278994582e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.415770 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.415733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/02e89a48-96ff-4fde-8ff3-9d278994582e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.415770 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.415761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj55b\" (UniqueName: \"kubernetes.io/projected/02e89a48-96ff-4fde-8ff3-9d278994582e-kube-api-access-qj55b\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.415985 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:23:43.415851    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 19 15:23:43.415985 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:23:43.415932    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e89a48-96ff-4fde-8ff3-9d278994582e-plugin-serving-cert podName:02e89a48-96ff-4fde-8ff3-9d278994582e nodeName:}" failed. No retries permitted until 2026-09-19 15:23:43.915912751 +0000 UTC m=+685.809476369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/02e89a48-96ff-4fde-8ff3-9d278994582e-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-btgkd" (UID: "02e89a48-96ff-4fde-8ff3-9d278994582e") : secret "plugin-serving-cert" not found
Sep 19 15:23:43.416357 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.416340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/02e89a48-96ff-4fde-8ff3-9d278994582e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.423760 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.423742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj55b\" (UniqueName: \"kubernetes.io/projected/02e89a48-96ff-4fde-8ff3-9d278994582e-kube-api-access-qj55b\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.919439 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.919402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/02e89a48-96ff-4fde-8ff3-9d278994582e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:43.921992 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:43.921956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/02e89a48-96ff-4fde-8ff3-9d278994582e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-btgkd\" (UID: \"02e89a48-96ff-4fde-8ff3-9d278994582e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:44.076386 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:44.076361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"
Sep 19 15:23:44.344224 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:44.344198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd"]
Sep 19 15:23:44.346723 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:23:44.346699    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02e89a48_96ff_4fde_8ff3_9d278994582e.slice/crio-c0232afc034aaa32553623145e155f22794dd45ebee2c57150f5af42a5595d2d WatchSource:0}: Error finding container c0232afc034aaa32553623145e155f22794dd45ebee2c57150f5af42a5595d2d: Status 404 returned error can't find the container with id c0232afc034aaa32553623145e155f22794dd45ebee2c57150f5af42a5595d2d
Sep 19 15:23:44.347931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:44.347913    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 15:23:44.549047 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:44.549007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd" event={"ID":"02e89a48-96ff-4fde-8ff3-9d278994582e","Type":"ContainerStarted","Data":"c0232afc034aaa32553623145e155f22794dd45ebee2c57150f5af42a5595d2d"}
Sep 19 15:23:49.572876 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:49.572839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd" event={"ID":"02e89a48-96ff-4fde-8ff3-9d278994582e","Type":"ContainerStarted","Data":"9a688e3c669923a23b0f40952324954db9991335e30638a05603d4b9acdbab10"}
Sep 19 15:23:49.587211 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:23:49.587160    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-btgkd" podStartSLOduration=2.12176859 podStartE2EDuration="6.587139817s" totalImagesPullingTime="4.465371227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:23:49.585347351 +0000 UTC m=+691.478910990" watchObservedRunningTime="2026-09-19 15:23:49.587139817 +0000 UTC m=+691.480703455"
Sep 19 15:24:38.012171 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.012134    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-prkvj"]
Sep 19 15:24:38.015086 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.015068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj"
Sep 19 15:24:38.017119 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.017089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q5lnm\""
Sep 19 15:24:38.022370 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.022317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-prkvj"]
Sep 19 15:24:38.136183 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.136153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8dt\" (UniqueName: \"kubernetes.io/projected/df16cf60-9fcd-416f-8a81-268a9ea9812d-kube-api-access-5g8dt\") pod \"authorino-7bcfd8f9cd-prkvj\" (UID: \"df16cf60-9fcd-416f-8a81-268a9ea9812d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj"
Sep 19 15:24:38.237110 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.237087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8dt\" (UniqueName: \"kubernetes.io/projected/df16cf60-9fcd-416f-8a81-268a9ea9812d-kube-api-access-5g8dt\") pod \"authorino-7bcfd8f9cd-prkvj\" (UID: \"df16cf60-9fcd-416f-8a81-268a9ea9812d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj"
Sep 19 15:24:38.244266 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.244246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8dt\" (UniqueName: \"kubernetes.io/projected/df16cf60-9fcd-416f-8a81-268a9ea9812d-kube-api-access-5g8dt\") pod \"authorino-7bcfd8f9cd-prkvj\" (UID: \"df16cf60-9fcd-416f-8a81-268a9ea9812d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj"
Sep 19 15:24:38.260177 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.260149    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-rmgwj"]
Sep 19 15:24:38.263325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.263289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rmgwj"
Sep 19 15:24:38.268909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.268886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rmgwj"]
Sep 19 15:24:38.324572 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.324549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj"
Sep 19 15:24:38.338465 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.338444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6h8\" (UniqueName: \"kubernetes.io/projected/d29e352c-8671-4a7d-a644-182354f5112c-kube-api-access-mh6h8\") pod \"authorino-59c974bd48-rmgwj\" (UID: \"d29e352c-8671-4a7d-a644-182354f5112c\") " pod="kuadrant-system/authorino-59c974bd48-rmgwj"
Sep 19 15:24:38.439716 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.439691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6h8\" (UniqueName: \"kubernetes.io/projected/d29e352c-8671-4a7d-a644-182354f5112c-kube-api-access-mh6h8\") pod \"authorino-59c974bd48-rmgwj\" (UID: \"d29e352c-8671-4a7d-a644-182354f5112c\") " pod="kuadrant-system/authorino-59c974bd48-rmgwj"
Sep 19 15:24:38.447210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.447181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6h8\" (UniqueName: \"kubernetes.io/projected/d29e352c-8671-4a7d-a644-182354f5112c-kube-api-access-mh6h8\") pod \"authorino-59c974bd48-rmgwj\" (UID: \"d29e352c-8671-4a7d-a644-182354f5112c\") " pod="kuadrant-system/authorino-59c974bd48-rmgwj"
Sep 19 15:24:38.573224 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.573174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rmgwj"
Sep 19 15:24:38.589715 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.589690    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-prkvj"]
Sep 19 15:24:38.591371 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:24:38.591348    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf16cf60_9fcd_416f_8a81_268a9ea9812d.slice/crio-5e6772e0a7a9734488bc516b9ae9462eb95a68d1c970d6d41b1cf68355e83100 WatchSource:0}: Error finding container 5e6772e0a7a9734488bc516b9ae9462eb95a68d1c970d6d41b1cf68355e83100: Status 404 returned error can't find the container with id 5e6772e0a7a9734488bc516b9ae9462eb95a68d1c970d6d41b1cf68355e83100
Sep 19 15:24:38.768793 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.768762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj" event={"ID":"df16cf60-9fcd-416f-8a81-268a9ea9812d","Type":"ContainerStarted","Data":"5e6772e0a7a9734488bc516b9ae9462eb95a68d1c970d6d41b1cf68355e83100"}
Sep 19 15:24:38.847727 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:38.847652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rmgwj"]
Sep 19 15:24:38.850244 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:24:38.850222    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd29e352c_8671_4a7d_a644_182354f5112c.slice/crio-7e80fcfd6badff6251a7bb0cf42348cf97e6a005d0166632fcee6a7cccfdf417 WatchSource:0}: Error finding container 7e80fcfd6badff6251a7bb0cf42348cf97e6a005d0166632fcee6a7cccfdf417: Status 404 returned error can't find the container with id 7e80fcfd6badff6251a7bb0cf42348cf97e6a005d0166632fcee6a7cccfdf417
Sep 19 15:24:39.775760 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:39.775699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rmgwj" event={"ID":"d29e352c-8671-4a7d-a644-182354f5112c","Type":"ContainerStarted","Data":"7e80fcfd6badff6251a7bb0cf42348cf97e6a005d0166632fcee6a7cccfdf417"}
Sep 19 15:24:41.792581 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:41.792543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj" event={"ID":"df16cf60-9fcd-416f-8a81-268a9ea9812d","Type":"ContainerStarted","Data":"df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063"}
Sep 19 15:24:41.794126 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:41.794101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rmgwj" event={"ID":"d29e352c-8671-4a7d-a644-182354f5112c","Type":"ContainerStarted","Data":"f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55"}
Sep 19 15:24:41.805234 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:41.805191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj" podStartSLOduration=1.7567486300000001 podStartE2EDuration="4.805176761s" totalImagesPullingTime="3.048428131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:24:41.804321343 +0000 UTC m=+743.697884981" watchObservedRunningTime="2026-09-19 15:24:41.805176761 +0000 UTC m=+743.698740401"
Sep 19 15:24:41.817171 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:41.817119    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-rmgwj" podStartSLOduration=1.300182886 podStartE2EDuration="3.817099965s" totalImagesPullingTime="2.516917079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:24:41.816012835 +0000 UTC m=+743.709576476" watchObservedRunningTime="2026-09-19 15:24:41.817099965 +0000 UTC m=+743.710663605"
Sep 19 15:24:41.839296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:41.839269    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-prkvj"]
Sep 19 15:24:43.801584 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:43.801545    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj" podUID="df16cf60-9fcd-416f-8a81-268a9ea9812d" containerName="authorino" containerID="cri-o://df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063" gracePeriod=30
Sep 19 15:24:44.030855 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.030835    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj"
Sep 19 15:24:44.083174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.083110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g8dt\" (UniqueName: \"kubernetes.io/projected/df16cf60-9fcd-416f-8a81-268a9ea9812d-kube-api-access-5g8dt\") pod \"df16cf60-9fcd-416f-8a81-268a9ea9812d\" (UID: \"df16cf60-9fcd-416f-8a81-268a9ea9812d\") "
Sep 19 15:24:44.085234 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.085210    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df16cf60-9fcd-416f-8a81-268a9ea9812d-kube-api-access-5g8dt" (OuterVolumeSpecName: "kube-api-access-5g8dt") pod "df16cf60-9fcd-416f-8a81-268a9ea9812d" (UID: "df16cf60-9fcd-416f-8a81-268a9ea9812d"). InnerVolumeSpecName "kube-api-access-5g8dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:24:44.184228 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.184210    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g8dt\" (UniqueName: \"kubernetes.io/projected/df16cf60-9fcd-416f-8a81-268a9ea9812d-kube-api-access-5g8dt\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:24:44.806208 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.806180    2579 generic.go:356] "Generic (PLEG): container finished" podID="df16cf60-9fcd-416f-8a81-268a9ea9812d" containerID="df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063" exitCode=0
Sep 19 15:24:44.806572 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.806224    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj"
Sep 19 15:24:44.806572 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.806237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj" event={"ID":"df16cf60-9fcd-416f-8a81-268a9ea9812d","Type":"ContainerDied","Data":"df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063"}
Sep 19 15:24:44.806572 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.806267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-prkvj" event={"ID":"df16cf60-9fcd-416f-8a81-268a9ea9812d","Type":"ContainerDied","Data":"5e6772e0a7a9734488bc516b9ae9462eb95a68d1c970d6d41b1cf68355e83100"}
Sep 19 15:24:44.806572 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.806282    2579 scope.go:117] "RemoveContainer" containerID="df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063"
Sep 19 15:24:44.813826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.813811    2579 scope.go:117] "RemoveContainer" containerID="df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063"
Sep 19 15:24:44.814077 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:24:44.814059    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063\": container with ID starting with df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063 not found: ID does not exist" containerID="df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063"
Sep 19 15:24:44.814123 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.814084    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063"} err="failed to get container status \"df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063\": rpc error: code = NotFound desc = could not find container \"df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063\": container with ID starting with df86eba751e77f0d459442f71ceac8b3603018cfd89e0f7f61d28c8aa805c063 not found: ID does not exist"
Sep 19 15:24:44.820816 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.820723    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-prkvj"]
Sep 19 15:24:44.822051 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:44.822033    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-prkvj"]
Sep 19 15:24:46.169955 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:24:46.169914    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df16cf60-9fcd-416f-8a81-268a9ea9812d" path="/var/lib/kubelet/pods/df16cf60-9fcd-416f-8a81-268a9ea9812d/volumes"
Sep 19 15:25:08.326151 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.326118    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-lx9fn"]
Sep 19 15:25:08.326564 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.326431    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df16cf60-9fcd-416f-8a81-268a9ea9812d" containerName="authorino"
Sep 19 15:25:08.326564 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.326444    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df16cf60-9fcd-416f-8a81-268a9ea9812d" containerName="authorino"
Sep 19 15:25:08.326564 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.326493    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df16cf60-9fcd-416f-8a81-268a9ea9812d" containerName="authorino"
Sep 19 15:25:08.332695 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.332644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lx9fn"
Sep 19 15:25:08.335390 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.335370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-lx9fn"]
Sep 19 15:25:08.471390 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.471355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9kn\" (UniqueName: \"kubernetes.io/projected/db3b86fa-f686-4450-b26c-48ae1849283b-kube-api-access-tk9kn\") pod \"authorino-557f876f69-lx9fn\" (UID: \"db3b86fa-f686-4450-b26c-48ae1849283b\") " pod="kuadrant-system/authorino-557f876f69-lx9fn"
Sep 19 15:25:08.552934 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.552895    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lx9fn"]
Sep 19 15:25:08.553188 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:25:08.553164    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tk9kn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-lx9fn" podUID="db3b86fa-f686-4450-b26c-48ae1849283b"
Sep 19 15:25:08.572498 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.572466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9kn\" (UniqueName: \"kubernetes.io/projected/db3b86fa-f686-4450-b26c-48ae1849283b-kube-api-access-tk9kn\") pod \"authorino-557f876f69-lx9fn\" (UID: \"db3b86fa-f686-4450-b26c-48ae1849283b\") " pod="kuadrant-system/authorino-557f876f69-lx9fn"
Sep 19 15:25:08.577697 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.577629    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-677945dc8d-bvp2k"]
Sep 19 15:25:08.580909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.580886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9kn\" (UniqueName: \"kubernetes.io/projected/db3b86fa-f686-4450-b26c-48ae1849283b-kube-api-access-tk9kn\") pod \"authorino-557f876f69-lx9fn\" (UID: \"db3b86fa-f686-4450-b26c-48ae1849283b\") " pod="kuadrant-system/authorino-557f876f69-lx9fn"
Sep 19 15:25:08.581406 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.581377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-677945dc8d-bvp2k"
Sep 19 15:25:08.586912 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.586887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-677945dc8d-bvp2k"]
Sep 19 15:25:08.673926 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.673894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp626\" (UniqueName: \"kubernetes.io/projected/1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b-kube-api-access-bp626\") pod \"authorino-677945dc8d-bvp2k\" (UID: \"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b\") " pod="kuadrant-system/authorino-677945dc8d-bvp2k"
Sep 19 15:25:08.775247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.775218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp626\" (UniqueName: \"kubernetes.io/projected/1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b-kube-api-access-bp626\") pod \"authorino-677945dc8d-bvp2k\" (UID: \"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b\") " pod="kuadrant-system/authorino-677945dc8d-bvp2k"
Sep 19 15:25:08.782595 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.782569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp626\" (UniqueName: \"kubernetes.io/projected/1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b-kube-api-access-bp626\") pod \"authorino-677945dc8d-bvp2k\" (UID: \"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b\") " pod="kuadrant-system/authorino-677945dc8d-bvp2k"
Sep 19 15:25:08.896888 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.896795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-677945dc8d-bvp2k"
Sep 19 15:25:08.900492 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.900469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lx9fn"
Sep 19 15:25:08.908530 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:08.908512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lx9fn"
Sep 19 15:25:09.077985 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.077954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk9kn\" (UniqueName: \"kubernetes.io/projected/db3b86fa-f686-4450-b26c-48ae1849283b-kube-api-access-tk9kn\") pod \"db3b86fa-f686-4450-b26c-48ae1849283b\" (UID: \"db3b86fa-f686-4450-b26c-48ae1849283b\") "
Sep 19 15:25:09.080133 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.080110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3b86fa-f686-4450-b26c-48ae1849283b-kube-api-access-tk9kn" (OuterVolumeSpecName: "kube-api-access-tk9kn") pod "db3b86fa-f686-4450-b26c-48ae1849283b" (UID: "db3b86fa-f686-4450-b26c-48ae1849283b"). InnerVolumeSpecName "kube-api-access-tk9kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:25:09.162867 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.162839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-677945dc8d-bvp2k"]
Sep 19 15:25:09.165382 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:25:09.165355    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d5771f7_bbdb_4b8b_acbc_cbff4f73a09b.slice/crio-057912867d6d12004ddc2c788891530d6009bb3b4a8eb1e725b4205e032c17f7 WatchSource:0}: Error finding container 057912867d6d12004ddc2c788891530d6009bb3b4a8eb1e725b4205e032c17f7: Status 404 returned error can't find the container with id 057912867d6d12004ddc2c788891530d6009bb3b4a8eb1e725b4205e032c17f7
Sep 19 15:25:09.178944 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.178926    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9kn\" (UniqueName: \"kubernetes.io/projected/db3b86fa-f686-4450-b26c-48ae1849283b-kube-api-access-tk9kn\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:25:09.906580 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.906483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-677945dc8d-bvp2k" event={"ID":"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b","Type":"ContainerStarted","Data":"1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c"}
Sep 19 15:25:09.906580 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.906529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-677945dc8d-bvp2k" event={"ID":"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b","Type":"ContainerStarted","Data":"057912867d6d12004ddc2c788891530d6009bb3b4a8eb1e725b4205e032c17f7"}
Sep 19 15:25:09.906580 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.906560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lx9fn"
Sep 19 15:25:09.920233 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.920195    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-677945dc8d-bvp2k" podStartSLOduration=1.528836109 podStartE2EDuration="1.920179801s" totalImagesPullingTime="391.343692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:25:09.919524282 +0000 UTC m=+771.813087923" watchObservedRunningTime="2026-09-19 15:25:09.920179801 +0000 UTC m=+771.813743444"
Sep 19 15:25:09.940789 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.940762    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lx9fn"]
Sep 19 15:25:09.947129 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.947101    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rmgwj"]
Sep 19 15:25:09.947345 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.947320    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-rmgwj" podUID="d29e352c-8671-4a7d-a644-182354f5112c" containerName="authorino" containerID="cri-o://f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55" gracePeriod=30
Sep 19 15:25:09.948495 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:09.948477    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-lx9fn"]
Sep 19 15:25:10.010640 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.010588    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cc55bcd78-nm4fn"]
Sep 19 15:25:10.013670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.013649    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:25:10.015697 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.015677    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 19 15:25:10.019541 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.019518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cc55bcd78-nm4fn"]
Sep 19 15:25:10.170710 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.170684    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db3b86fa-f686-4450-b26c-48ae1849283b" path="/var/lib/kubelet/pods/db3b86fa-f686-4450-b26c-48ae1849283b/volumes"
Sep 19 15:25:10.186592 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.186569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-tls-cert\") pod \"authorino-6cc55bcd78-nm4fn\" (UID: \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\") " pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:25:10.186737 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.186631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5nm\" (UniqueName: \"kubernetes.io/projected/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-kube-api-access-xs5nm\") pod \"authorino-6cc55bcd78-nm4fn\" (UID: \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\") " pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:25:10.189082 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.189066    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rmgwj"
Sep 19 15:25:10.287010 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.286984    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh6h8\" (UniqueName: \"kubernetes.io/projected/d29e352c-8671-4a7d-a644-182354f5112c-kube-api-access-mh6h8\") pod \"d29e352c-8671-4a7d-a644-182354f5112c\" (UID: \"d29e352c-8671-4a7d-a644-182354f5112c\") "
Sep 19 15:25:10.287173 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.287105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-tls-cert\") pod \"authorino-6cc55bcd78-nm4fn\" (UID: \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\") " pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:25:10.287173 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.287143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5nm\" (UniqueName: \"kubernetes.io/projected/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-kube-api-access-xs5nm\") pod \"authorino-6cc55bcd78-nm4fn\" (UID: \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\") " pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:25:10.289400 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.289363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29e352c-8671-4a7d-a644-182354f5112c-kube-api-access-mh6h8" (OuterVolumeSpecName: "kube-api-access-mh6h8") pod "d29e352c-8671-4a7d-a644-182354f5112c" (UID: "d29e352c-8671-4a7d-a644-182354f5112c"). InnerVolumeSpecName "kube-api-access-mh6h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:25:10.289942 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.289922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-tls-cert\") pod \"authorino-6cc55bcd78-nm4fn\" (UID: \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\") " pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:25:10.294489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.294467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5nm\" (UniqueName: \"kubernetes.io/projected/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-kube-api-access-xs5nm\") pod \"authorino-6cc55bcd78-nm4fn\" (UID: \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\") " pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:25:10.324409 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.324387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:25:10.388520 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.388491    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6h8\" (UniqueName: \"kubernetes.io/projected/d29e352c-8671-4a7d-a644-182354f5112c-kube-api-access-mh6h8\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:25:10.592013 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.591975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cc55bcd78-nm4fn"]
Sep 19 15:25:10.595026 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:25:10.594997    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2c5ba0_643c_4bd1_8d01_d44cf8b37075.slice/crio-68abecac858cae4d536853ceba9a003b7cc22538b806f0dd50a86dabd8dd65f3 WatchSource:0}: Error finding container 68abecac858cae4d536853ceba9a003b7cc22538b806f0dd50a86dabd8dd65f3: Status 404 returned error can't find the container with id 68abecac858cae4d536853ceba9a003b7cc22538b806f0dd50a86dabd8dd65f3
Sep 19 15:25:10.912153 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.912072    2579 generic.go:356] "Generic (PLEG): container finished" podID="d29e352c-8671-4a7d-a644-182354f5112c" containerID="f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55" exitCode=0
Sep 19 15:25:10.912153 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.912114    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rmgwj"
Sep 19 15:25:10.912648 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.912159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rmgwj" event={"ID":"d29e352c-8671-4a7d-a644-182354f5112c","Type":"ContainerDied","Data":"f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55"}
Sep 19 15:25:10.912648 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.912201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rmgwj" event={"ID":"d29e352c-8671-4a7d-a644-182354f5112c","Type":"ContainerDied","Data":"7e80fcfd6badff6251a7bb0cf42348cf97e6a005d0166632fcee6a7cccfdf417"}
Sep 19 15:25:10.912648 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.912229    2579 scope.go:117] "RemoveContainer" containerID="f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55"
Sep 19 15:25:10.913488 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.913467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn" event={"ID":"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075","Type":"ContainerStarted","Data":"68abecac858cae4d536853ceba9a003b7cc22538b806f0dd50a86dabd8dd65f3"}
Sep 19 15:25:10.921160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.921145    2579 scope.go:117] "RemoveContainer" containerID="f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55"
Sep 19 15:25:10.921406 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:25:10.921392    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55\": container with ID starting with f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55 not found: ID does not exist" containerID="f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55"
Sep 19 15:25:10.921452 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.921413    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55"} err="failed to get container status \"f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55\": rpc error: code = NotFound desc = could not find container \"f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55\": container with ID starting with f8ea41db7dbbb2bb2d5b1afa6b4231d2b29e9eb61ed7acb44e9949d5ca466f55 not found: ID does not exist"
Sep 19 15:25:10.930792 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.930771    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rmgwj"]
Sep 19 15:25:10.938685 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:10.935627    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rmgwj"]
Sep 19 15:25:11.920880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:11.920848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn" event={"ID":"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075","Type":"ContainerStarted","Data":"3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9"}
Sep 19 15:25:11.934864 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:11.934815    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn" podStartSLOduration=2.427582186 podStartE2EDuration="2.934800373s" totalImagesPullingTime="507.218187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:25:11.933691522 +0000 UTC m=+773.827255161" watchObservedRunningTime="2026-09-19 15:25:11.934800373 +0000 UTC m=+773.828364012"
Sep 19 15:25:11.957835 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:11.957804    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-677945dc8d-bvp2k"]
Sep 19 15:25:11.958002 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:11.957973    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-677945dc8d-bvp2k" podUID="1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b" containerName="authorino" containerID="cri-o://1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c" gracePeriod=30
Sep 19 15:25:12.171484 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.171395    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29e352c-8671-4a7d-a644-182354f5112c" path="/var/lib/kubelet/pods/d29e352c-8671-4a7d-a644-182354f5112c/volumes"
Sep 19 15:25:12.198033 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.198013    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-677945dc8d-bvp2k"
Sep 19 15:25:12.304217 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.304185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp626\" (UniqueName: \"kubernetes.io/projected/1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b-kube-api-access-bp626\") pod \"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b\" (UID: \"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b\") "
Sep 19 15:25:12.306325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.306300    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b-kube-api-access-bp626" (OuterVolumeSpecName: "kube-api-access-bp626") pod "1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b" (UID: "1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b"). InnerVolumeSpecName "kube-api-access-bp626". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:25:12.405429 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.405405    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp626\" (UniqueName: \"kubernetes.io/projected/1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b-kube-api-access-bp626\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:25:12.926675 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.926640    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b" containerID="1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c" exitCode=0
Sep 19 15:25:12.926675 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.926685    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-677945dc8d-bvp2k"
Sep 19 15:25:12.927122 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.926721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-677945dc8d-bvp2k" event={"ID":"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b","Type":"ContainerDied","Data":"1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c"}
Sep 19 15:25:12.927122 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.926761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-677945dc8d-bvp2k" event={"ID":"1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b","Type":"ContainerDied","Data":"057912867d6d12004ddc2c788891530d6009bb3b4a8eb1e725b4205e032c17f7"}
Sep 19 15:25:12.927122 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.926779    2579 scope.go:117] "RemoveContainer" containerID="1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c"
Sep 19 15:25:12.935005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.934986    2579 scope.go:117] "RemoveContainer" containerID="1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c"
Sep 19 15:25:12.935266 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:25:12.935247    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c\": container with ID starting with 1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c not found: ID does not exist" containerID="1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c"
Sep 19 15:25:12.935322 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.935274    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c"} err="failed to get container status \"1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c\": rpc error: code = NotFound desc = could not find container \"1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c\": container with ID starting with 1887bc7b7628d3b717d68dc975b1b463ca186b05a84b1ae0e4ce7ae83ab68b5c not found: ID does not exist"
Sep 19 15:25:12.945437 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.945417    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-677945dc8d-bvp2k"]
Sep 19 15:25:12.948313 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:12.948294    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-677945dc8d-bvp2k"]
Sep 19 15:25:14.169781 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:14.169751    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b" path="/var/lib/kubelet/pods/1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b/volumes"
Sep 19 15:25:52.093808 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.093735    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"]
Sep 19 15:25:52.094221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.094033    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b" containerName="authorino"
Sep 19 15:25:52.094221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.094044    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b" containerName="authorino"
Sep 19 15:25:52.094221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.094054    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29e352c-8671-4a7d-a644-182354f5112c" containerName="authorino"
Sep 19 15:25:52.094221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.094059    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29e352c-8671-4a7d-a644-182354f5112c" containerName="authorino"
Sep 19 15:25:52.094221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.094115    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5771f7-bbdb-4b8b-acbc-cbff4f73a09b" containerName="authorino"
Sep 19 15:25:52.094221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.094125    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29e352c-8671-4a7d-a644-182354f5112c" containerName="authorino"
Sep 19 15:25:52.097091 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.097072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:52.098976 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.098956    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jkrrr\""
Sep 19 15:25:52.099061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.098992    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 19 15:25:52.107998 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.107975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"]
Sep 19 15:25:52.202030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.201990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tsqv\" (UniqueName: \"kubernetes.io/projected/d0684ca3-3228-4473-82e5-fd5397562502-kube-api-access-7tsqv\") pod \"payload-processing-5b54b7fbb7-zzvr5\" (UID: \"d0684ca3-3228-4473-82e5-fd5397562502\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:52.202184 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.202121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0684ca3-3228-4473-82e5-fd5397562502-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zzvr5\" (UID: \"d0684ca3-3228-4473-82e5-fd5397562502\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:52.303533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.303493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0684ca3-3228-4473-82e5-fd5397562502-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zzvr5\" (UID: \"d0684ca3-3228-4473-82e5-fd5397562502\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:52.303888 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.303866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tsqv\" (UniqueName: \"kubernetes.io/projected/d0684ca3-3228-4473-82e5-fd5397562502-kube-api-access-7tsqv\") pod \"payload-processing-5b54b7fbb7-zzvr5\" (UID: \"d0684ca3-3228-4473-82e5-fd5397562502\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:52.304431 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.304406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0684ca3-3228-4473-82e5-fd5397562502-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zzvr5\" (UID: \"d0684ca3-3228-4473-82e5-fd5397562502\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:52.315753 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.315721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tsqv\" (UniqueName: \"kubernetes.io/projected/d0684ca3-3228-4473-82e5-fd5397562502-kube-api-access-7tsqv\") pod \"payload-processing-5b54b7fbb7-zzvr5\" (UID: \"d0684ca3-3228-4473-82e5-fd5397562502\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:52.407740 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.407653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:52.677836 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:52.677814    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"]
Sep 19 15:25:52.680260 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:25:52.680233    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0684ca3_3228_4473_82e5_fd5397562502.slice/crio-55ae151d621039d1a588561baf6f53edc6e605185bbd25d81f81960c8e148a6c WatchSource:0}: Error finding container 55ae151d621039d1a588561baf6f53edc6e605185bbd25d81f81960c8e148a6c: Status 404 returned error can't find the container with id 55ae151d621039d1a588561baf6f53edc6e605185bbd25d81f81960c8e148a6c
Sep 19 15:25:53.086741 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:53.086688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5" event={"ID":"d0684ca3-3228-4473-82e5-fd5397562502","Type":"ContainerStarted","Data":"55ae151d621039d1a588561baf6f53edc6e605185bbd25d81f81960c8e148a6c"}
Sep 19 15:25:54.093397 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:54.093357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5" event={"ID":"d0684ca3-3228-4473-82e5-fd5397562502","Type":"ContainerStarted","Data":"e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac"}
Sep 19 15:25:54.093788 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:54.093425    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:25:54.108054 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:25:54.108010    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5" podStartSLOduration=0.897612978 podStartE2EDuration="2.107995764s" totalImagesPullingTime="1.210382786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:25:54.106332277 +0000 UTC m=+815.999895918" watchObservedRunningTime="2026-09-19 15:25:54.107995764 +0000 UTC m=+816.001559404"
Sep 19 15:26:05.100384 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:26:05.100350    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:27:17.831457 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.831425    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"]
Sep 19 15:27:17.834876 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.834853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:17.838982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.838963    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 19 15:27:17.839680 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.839646    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mqcwz\""
Sep 19 15:27:17.839680 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.839655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 19 15:27:17.839842 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.839714    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 19 15:27:17.844042 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.844020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"]
Sep 19 15:27:17.904123 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.904091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcwhx\" (UniqueName: \"kubernetes.io/projected/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-kube-api-access-bcwhx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:17.904234 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.904167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:17.904234 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.904208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:17.904328 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.904270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:17.904380 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.904322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:17.904439 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:17.904380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.004953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.004925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.005050 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.004969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.005050 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.004996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.005159 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.005049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcwhx\" (UniqueName: \"kubernetes.io/projected/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-kube-api-access-bcwhx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.005159 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.005081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.005159 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.005119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.005300 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.005253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.005356 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.005321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.005443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.005426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.007351 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.007327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.007632 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.007597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.012496 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.012474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcwhx\" (UniqueName: \"kubernetes.io/projected/68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7-kube-api-access-bcwhx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm\" (UID: \"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.145744 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.145697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:18.415661 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.415637    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"]
Sep 19 15:27:18.418015 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:27:18.417980    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ebd6b7_fe5b_4868_9fb9_fd22d8b000a7.slice/crio-19a9d93538b28adead728dab12eac0962b566c2355fc14c41638155a67c1ea94 WatchSource:0}: Error finding container 19a9d93538b28adead728dab12eac0962b566c2355fc14c41638155a67c1ea94: Status 404 returned error can't find the container with id 19a9d93538b28adead728dab12eac0962b566c2355fc14c41638155a67c1ea94
Sep 19 15:27:18.427461 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:18.427436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm" event={"ID":"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7","Type":"ContainerStarted","Data":"19a9d93538b28adead728dab12eac0962b566c2355fc14c41638155a67c1ea94"}
Sep 19 15:27:20.124676 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:20.124651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:27:20.125638 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:20.125620    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:27:20.437359 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:20.437323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm" event={"ID":"68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7","Type":"ContainerStarted","Data":"a5a4f3a62f53aaceaedb8ab7800955c0b034526515a75e074bd82c8467059dce"}
Sep 19 15:27:20.453433 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:20.453389    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm" podStartSLOduration=2.476189491 podStartE2EDuration="3.453373836s" totalImagesPullingTime="977.184345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:27:20.451366703 +0000 UTC m=+902.344930345" watchObservedRunningTime="2026-09-19 15:27:20.453373836 +0000 UTC m=+902.346937475"
Sep 19 15:27:28.145869 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:28.145833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:28.148194 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:28.145900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:28.157568 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:28.157548    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:28.480072 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:28.480051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm"
Sep 19 15:27:33.743432 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.743397    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"]
Sep 19 15:27:33.746634 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.746615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.749513 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.749489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 19 15:27:33.750974 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.750950    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"]
Sep 19 15:27:33.817872 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.817845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.818011 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.817893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkn4z\" (UniqueName: \"kubernetes.io/projected/bb9f2be9-d264-4d21-8255-bbcdf96e9152-kube-api-access-pkn4z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.818011 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.817972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.818126 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.818023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.818126 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.818049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9f2be9-d264-4d21-8255-bbcdf96e9152-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.818126 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.818077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919183 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkn4z\" (UniqueName: \"kubernetes.io/projected/bb9f2be9-d264-4d21-8255-bbcdf96e9152-kube-api-access-pkn4z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919183 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9f2be9-d264-4d21-8255-bbcdf96e9152-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919457 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919511 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.919638 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.919593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.921410 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.921390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9f2be9-d264-4d21-8255-bbcdf96e9152-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.921707 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.921692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9f2be9-d264-4d21-8255-bbcdf96e9152-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:33.926270 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:33.926252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkn4z\" (UniqueName: \"kubernetes.io/projected/bb9f2be9-d264-4d21-8255-bbcdf96e9152-kube-api-access-pkn4z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6crrt\" (UID: \"bb9f2be9-d264-4d21-8255-bbcdf96e9152\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:34.056825 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:34.056776    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:34.322089 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:34.322059    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"]
Sep 19 15:27:34.324031 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:27:34.324007    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb9f2be9_d264_4d21_8255_bbcdf96e9152.slice/crio-3fde5bfce3b2a14cb9ce028eccb3432aa82125f12e9873402c6bc037dd53e030 WatchSource:0}: Error finding container 3fde5bfce3b2a14cb9ce028eccb3432aa82125f12e9873402c6bc037dd53e030: Status 404 returned error can't find the container with id 3fde5bfce3b2a14cb9ce028eccb3432aa82125f12e9873402c6bc037dd53e030
Sep 19 15:27:34.495239 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:34.495211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt" event={"ID":"bb9f2be9-d264-4d21-8255-bbcdf96e9152","Type":"ContainerStarted","Data":"3fde5bfce3b2a14cb9ce028eccb3432aa82125f12e9873402c6bc037dd53e030"}
Sep 19 15:27:35.501345 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:35.501309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt" event={"ID":"bb9f2be9-d264-4d21-8255-bbcdf96e9152","Type":"ContainerStarted","Data":"c5e63463b853baf36331b994f505b6fbd0e7cfb1f01a03c3b4d06ab936e97e45"}
Sep 19 15:27:35.516231 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:35.516182    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt" podStartSLOduration=2.351227655 podStartE2EDuration="2.516167013s" totalImagesPullingTime="164.939358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:27:35.515159412 +0000 UTC m=+917.408723052" watchObservedRunningTime="2026-09-19 15:27:35.516167013 +0000 UTC m=+917.409730661"
Sep 19 15:27:44.057324 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:44.057294    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:44.057696 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:44.057336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:44.069383 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:44.069361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:27:44.547251 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:27:44.547224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6crrt"
Sep 19 15:28:58.091411 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.091328    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85db744fc5-vhd4x"]
Sep 19 15:28:58.094528 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.094507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85db744fc5-vhd4x"
Sep 19 15:28:58.101675 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.101654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85db744fc5-vhd4x"]
Sep 19 15:28:58.132279 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.132258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/42419a3e-7e1a-4f7e-a111-1b0093b24e66-tls-cert\") pod \"authorino-85db744fc5-vhd4x\" (UID: \"42419a3e-7e1a-4f7e-a111-1b0093b24e66\") " pod="kuadrant-system/authorino-85db744fc5-vhd4x"
Sep 19 15:28:58.132394 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.132313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmf9t\" (UniqueName: \"kubernetes.io/projected/42419a3e-7e1a-4f7e-a111-1b0093b24e66-kube-api-access-vmf9t\") pod \"authorino-85db744fc5-vhd4x\" (UID: \"42419a3e-7e1a-4f7e-a111-1b0093b24e66\") " pod="kuadrant-system/authorino-85db744fc5-vhd4x"
Sep 19 15:28:58.232900 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.232878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/42419a3e-7e1a-4f7e-a111-1b0093b24e66-tls-cert\") pod \"authorino-85db744fc5-vhd4x\" (UID: \"42419a3e-7e1a-4f7e-a111-1b0093b24e66\") " pod="kuadrant-system/authorino-85db744fc5-vhd4x"
Sep 19 15:28:58.233026 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.232945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmf9t\" (UniqueName: \"kubernetes.io/projected/42419a3e-7e1a-4f7e-a111-1b0093b24e66-kube-api-access-vmf9t\") pod \"authorino-85db744fc5-vhd4x\" (UID: \"42419a3e-7e1a-4f7e-a111-1b0093b24e66\") " pod="kuadrant-system/authorino-85db744fc5-vhd4x"
Sep 19 15:28:58.235497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.235472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/42419a3e-7e1a-4f7e-a111-1b0093b24e66-tls-cert\") pod \"authorino-85db744fc5-vhd4x\" (UID: \"42419a3e-7e1a-4f7e-a111-1b0093b24e66\") " pod="kuadrant-system/authorino-85db744fc5-vhd4x"
Sep 19 15:28:58.239683 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.239665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmf9t\" (UniqueName: \"kubernetes.io/projected/42419a3e-7e1a-4f7e-a111-1b0093b24e66-kube-api-access-vmf9t\") pod \"authorino-85db744fc5-vhd4x\" (UID: \"42419a3e-7e1a-4f7e-a111-1b0093b24e66\") " pod="kuadrant-system/authorino-85db744fc5-vhd4x"
Sep 19 15:28:58.403947 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.403897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85db744fc5-vhd4x"
Sep 19 15:28:58.671857 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.671835    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85db744fc5-vhd4x"]
Sep 19 15:28:58.673929 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:28:58.673904    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42419a3e_7e1a_4f7e_a111_1b0093b24e66.slice/crio-8c318fe84eecd3393cd185d7cdf0f361ffcb5fa2a1796f2a71c53cbe5287634a WatchSource:0}: Error finding container 8c318fe84eecd3393cd185d7cdf0f361ffcb5fa2a1796f2a71c53cbe5287634a: Status 404 returned error can't find the container with id 8c318fe84eecd3393cd185d7cdf0f361ffcb5fa2a1796f2a71c53cbe5287634a
Sep 19 15:28:58.675154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.675135    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 15:28:58.836073 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:58.836048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85db744fc5-vhd4x" event={"ID":"42419a3e-7e1a-4f7e-a111-1b0093b24e66","Type":"ContainerStarted","Data":"8c318fe84eecd3393cd185d7cdf0f361ffcb5fa2a1796f2a71c53cbe5287634a"}
Sep 19 15:28:59.842848 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:59.842812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85db744fc5-vhd4x" event={"ID":"42419a3e-7e1a-4f7e-a111-1b0093b24e66","Type":"ContainerStarted","Data":"b72d1704e03126806547d9e25933446d21dc02851106094b514c0a9333ff4201"}
Sep 19 15:28:59.857368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:59.857316    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85db744fc5-vhd4x" podStartSLOduration=1.235959983 podStartE2EDuration="1.857300736s" totalImagesPullingTime="621.340753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:28:59.85622161 +0000 UTC m=+1001.749785250" watchObservedRunningTime="2026-09-19 15:28:59.857300736 +0000 UTC m=+1001.750864376"
Sep 19 15:28:59.885941 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:59.885903    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cc55bcd78-nm4fn"]
Sep 19 15:28:59.886410 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:28:59.886169    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn" podUID="4c2c5ba0-643c-4bd1-8d01-d44cf8b37075" containerName="authorino" containerID="cri-o://3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9" gracePeriod=30
Sep 19 15:29:00.131225 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.131196    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:29:00.251293 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.251258    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs5nm\" (UniqueName: \"kubernetes.io/projected/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-kube-api-access-xs5nm\") pod \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\" (UID: \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\") "
Sep 19 15:29:00.251293 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.251297    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-tls-cert\") pod \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\" (UID: \"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075\") "
Sep 19 15:29:00.253486 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.253446    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-kube-api-access-xs5nm" (OuterVolumeSpecName: "kube-api-access-xs5nm") pod "4c2c5ba0-643c-4bd1-8d01-d44cf8b37075" (UID: "4c2c5ba0-643c-4bd1-8d01-d44cf8b37075"). InnerVolumeSpecName "kube-api-access-xs5nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:29:00.261129 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.261104    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4c2c5ba0-643c-4bd1-8d01-d44cf8b37075" (UID: "4c2c5ba0-643c-4bd1-8d01-d44cf8b37075"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:29:00.353008 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.352975    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5nm\" (UniqueName: \"kubernetes.io/projected/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-kube-api-access-xs5nm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:29:00.353225 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.353208    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075-tls-cert\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:29:00.851423 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.851386    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c2c5ba0-643c-4bd1-8d01-d44cf8b37075" containerID="3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9" exitCode=0
Sep 19 15:29:00.852590 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.852569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn"
Sep 19 15:29:00.864795 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.864758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn" event={"ID":"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075","Type":"ContainerDied","Data":"3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9"}
Sep 19 15:29:00.864968 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.864951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cc55bcd78-nm4fn" event={"ID":"4c2c5ba0-643c-4bd1-8d01-d44cf8b37075","Type":"ContainerDied","Data":"68abecac858cae4d536853ceba9a003b7cc22538b806f0dd50a86dabd8dd65f3"}
Sep 19 15:29:00.865149 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.865050    2579 scope.go:117] "RemoveContainer" containerID="3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9"
Sep 19 15:29:00.882716 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.882692    2579 scope.go:117] "RemoveContainer" containerID="3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9"
Sep 19 15:29:00.883280 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:29:00.883258    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9\": container with ID starting with 3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9 not found: ID does not exist" containerID="3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9"
Sep 19 15:29:00.883376 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.883292    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9"} err="failed to get container status \"3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9\": rpc error: code = NotFound desc = could not find container \"3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9\": container with ID starting with 3a7c8773de2fe6feba88a04a3a7e953de5cf8d87a73816901496edabe52d7ff9 not found: ID does not exist"
Sep 19 15:29:00.886571 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.886551    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cc55bcd78-nm4fn"]
Sep 19 15:29:00.889727 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:00.889705    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cc55bcd78-nm4fn"]
Sep 19 15:29:02.170380 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:02.170348    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c2c5ba0-643c-4bd1-8d01-d44cf8b37075" path="/var/lib/kubelet/pods/4c2c5ba0-643c-4bd1-8d01-d44cf8b37075/volumes"
Sep 19 15:29:26.970058 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:26.970026    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"]
Sep 19 15:29:26.970461 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:26.970274    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5" podUID="d0684ca3-3228-4473-82e5-fd5397562502" containerName="payload-processing" containerID="cri-o://e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac" gracePeriod=30
Sep 19 15:29:27.210834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.210814    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:29:27.354337 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.354275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0684ca3-3228-4473-82e5-fd5397562502-plugins-config-volume\") pod \"d0684ca3-3228-4473-82e5-fd5397562502\" (UID: \"d0684ca3-3228-4473-82e5-fd5397562502\") "
Sep 19 15:29:27.354337 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.354331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tsqv\" (UniqueName: \"kubernetes.io/projected/d0684ca3-3228-4473-82e5-fd5397562502-kube-api-access-7tsqv\") pod \"d0684ca3-3228-4473-82e5-fd5397562502\" (UID: \"d0684ca3-3228-4473-82e5-fd5397562502\") "
Sep 19 15:29:27.354703 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.354679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0684ca3-3228-4473-82e5-fd5397562502-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0684ca3-3228-4473-82e5-fd5397562502" (UID: "d0684ca3-3228-4473-82e5-fd5397562502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:29:27.356531 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.356508    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0684ca3-3228-4473-82e5-fd5397562502-kube-api-access-7tsqv" (OuterVolumeSpecName: "kube-api-access-7tsqv") pod "d0684ca3-3228-4473-82e5-fd5397562502" (UID: "d0684ca3-3228-4473-82e5-fd5397562502"). InnerVolumeSpecName "kube-api-access-7tsqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:29:27.455511 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.455490    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0684ca3-3228-4473-82e5-fd5397562502-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:29:27.455511 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.455511    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tsqv\" (UniqueName: \"kubernetes.io/projected/d0684ca3-3228-4473-82e5-fd5397562502-kube-api-access-7tsqv\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:29:27.962735 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.962711    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0684ca3-3228-4473-82e5-fd5397562502" containerID="e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac" exitCode=0
Sep 19 15:29:27.962851 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.962751    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"
Sep 19 15:29:27.962851 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.962808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5" event={"ID":"d0684ca3-3228-4473-82e5-fd5397562502","Type":"ContainerDied","Data":"e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac"}
Sep 19 15:29:27.962921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.962859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zzvr5" event={"ID":"d0684ca3-3228-4473-82e5-fd5397562502","Type":"ContainerDied","Data":"55ae151d621039d1a588561baf6f53edc6e605185bbd25d81f81960c8e148a6c"}
Sep 19 15:29:27.962921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.962881    2579 scope.go:117] "RemoveContainer" containerID="e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac"
Sep 19 15:29:27.971229 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.971068    2579 scope.go:117] "RemoveContainer" containerID="e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac"
Sep 19 15:29:27.971430 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:29:27.971316    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac\": container with ID starting with e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac not found: ID does not exist" containerID="e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac"
Sep 19 15:29:27.971430 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.971339    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac"} err="failed to get container status \"e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac\": rpc error: code = NotFound desc = could not find container \"e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac\": container with ID starting with e86582f54672859a0a6067074442ad089d8b6480ffe8646441b272ad61bb88ac not found: ID does not exist"
Sep 19 15:29:27.981501 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.981478    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"]
Sep 19 15:29:27.982889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:27.982871    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zzvr5"]
Sep 19 15:29:28.169818 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:29:28.169795    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0684ca3-3228-4473-82e5-fd5397562502" path="/var/lib/kubelet/pods/d0684ca3-3228-4473-82e5-fd5397562502/volumes"
Sep 19 15:30:00.133096 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.133061    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d"]
Sep 19 15:30:00.133553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.133376    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0684ca3-3228-4473-82e5-fd5397562502" containerName="payload-processing"
Sep 19 15:30:00.133553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.133390    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0684ca3-3228-4473-82e5-fd5397562502" containerName="payload-processing"
Sep 19 15:30:00.133553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.133410    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2c5ba0-643c-4bd1-8d01-d44cf8b37075" containerName="authorino"
Sep 19 15:30:00.133553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.133415    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2c5ba0-643c-4bd1-8d01-d44cf8b37075" containerName="authorino"
Sep 19 15:30:00.133553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.133468    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c2c5ba0-643c-4bd1-8d01-d44cf8b37075" containerName="authorino"
Sep 19 15:30:00.133553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.133475    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0684ca3-3228-4473-82e5-fd5397562502" containerName="payload-processing"
Sep 19 15:30:00.136579 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.136564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d"
Sep 19 15:30:00.138932 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.138906    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7znsl\""
Sep 19 15:30:00.140234 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.140207    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d"]
Sep 19 15:30:00.148069 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.148051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d"
Sep 19 15:30:00.411525 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:00.411500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d"]
Sep 19 15:30:00.413842 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:00.413809    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cc9e7c_8431_4de2_9ffa_848f8bad93ee.slice/crio-7c67cb4d60084522bd027acdea3a3fd311102b7b766150558a8c48df0401750f WatchSource:0}: Error finding container 7c67cb4d60084522bd027acdea3a3fd311102b7b766150558a8c48df0401750f: Status 404 returned error can't find the container with id 7c67cb4d60084522bd027acdea3a3fd311102b7b766150558a8c48df0401750f
Sep 19 15:30:01.102058 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:01.102028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d" event={"ID":"a5cc9e7c-8431-4de2-9ffa-848f8bad93ee","Type":"ContainerStarted","Data":"7c67cb4d60084522bd027acdea3a3fd311102b7b766150558a8c48df0401750f"}
Sep 19 15:30:03.113007 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:03.112973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d" event={"ID":"a5cc9e7c-8431-4de2-9ffa-848f8bad93ee","Type":"ContainerStarted","Data":"268cbee55ac275b7a34d57abcb18d861c7a42e1f731bc9c13b98f305dac90ed9"}
Sep 19 15:30:03.124576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:03.124522    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d" podStartSLOduration=0.811182385 podStartE2EDuration="3.124503516s" totalImagesPullingTime="2.313321131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:03.123488488 +0000 UTC m=+1065.017052127" watchObservedRunningTime="2026-09-19 15:30:03.124503516 +0000 UTC m=+1065.018067157"
Sep 19 15:30:04.119301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:04.119267    2579 generic.go:356] "Generic (PLEG): container finished" podID="a5cc9e7c-8431-4de2-9ffa-848f8bad93ee" containerID="268cbee55ac275b7a34d57abcb18d861c7a42e1f731bc9c13b98f305dac90ed9" exitCode=0
Sep 19 15:30:04.119674 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:04.119348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d" event={"ID":"a5cc9e7c-8431-4de2-9ffa-848f8bad93ee","Type":"ContainerDied","Data":"268cbee55ac275b7a34d57abcb18d861c7a42e1f731bc9c13b98f305dac90ed9"}
Sep 19 15:30:05.245805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:05.245779    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d"
Sep 19 15:30:06.129443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:06.129408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d" event={"ID":"a5cc9e7c-8431-4de2-9ffa-848f8bad93ee","Type":"ContainerDied","Data":"7c67cb4d60084522bd027acdea3a3fd311102b7b766150558a8c48df0401750f"}
Sep 19 15:30:06.129443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:06.129443    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29830530-jh86d"
Sep 19 15:30:06.129759 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:06.129461    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c67cb4d60084522bd027acdea3a3fd311102b7b766150558a8c48df0401750f"
Sep 19 15:30:21.986226 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:21.986191    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9301-3a3727"]
Sep 19 15:30:21.986568 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:21.986504    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5cc9e7c-8431-4de2-9ffa-848f8bad93ee" containerName="cleanup"
Sep 19 15:30:21.986568 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:21.986517    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5cc9e7c-8431-4de2-9ffa-848f8bad93ee" containerName="cleanup"
Sep 19 15:30:21.986659 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:21.986587    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5cc9e7c-8431-4de2-9ffa-848f8bad93ee" containerName="cleanup"
Sep 19 15:30:21.990670 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:21.990650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9301-3a3727"
Sep 19 15:30:21.992625 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:21.992586    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hj4r7\""
Sep 19 15:30:21.994590 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:21.994540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9301-3a3727"]
Sep 19 15:30:22.038139 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:22.038117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5bv\" (UniqueName: \"kubernetes.io/projected/e337557f-07ff-469e-adb1-49a4573d963f-kube-api-access-nl5bv\") pod \"test-curl-9301-3a3727\" (UID: \"e337557f-07ff-469e-adb1-49a4573d963f\") " pod="opendatahub/test-curl-9301-3a3727"
Sep 19 15:30:22.139395 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:22.139375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5bv\" (UniqueName: \"kubernetes.io/projected/e337557f-07ff-469e-adb1-49a4573d963f-kube-api-access-nl5bv\") pod \"test-curl-9301-3a3727\" (UID: \"e337557f-07ff-469e-adb1-49a4573d963f\") " pod="opendatahub/test-curl-9301-3a3727"
Sep 19 15:30:22.150495 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:22.150475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5bv\" (UniqueName: \"kubernetes.io/projected/e337557f-07ff-469e-adb1-49a4573d963f-kube-api-access-nl5bv\") pod \"test-curl-9301-3a3727\" (UID: \"e337557f-07ff-469e-adb1-49a4573d963f\") " pod="opendatahub/test-curl-9301-3a3727"
Sep 19 15:30:22.300581 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:22.300518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9301-3a3727"
Sep 19 15:30:22.564632 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:22.564548    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9301-3a3727"]
Sep 19 15:30:22.568080 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:22.568059    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode337557f_07ff_469e_adb1_49a4573d963f.slice/crio-5d81f96c631c056c3de15a4fd25aa2ec0f682c3b46e590589a8ea4e3ab659131 WatchSource:0}: Error finding container 5d81f96c631c056c3de15a4fd25aa2ec0f682c3b46e590589a8ea4e3ab659131: Status 404 returned error can't find the container with id 5d81f96c631c056c3de15a4fd25aa2ec0f682c3b46e590589a8ea4e3ab659131
Sep 19 15:30:23.202906 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:23.202871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9301-3a3727" event={"ID":"e337557f-07ff-469e-adb1-49a4573d963f","Type":"ContainerStarted","Data":"5d81f96c631c056c3de15a4fd25aa2ec0f682c3b46e590589a8ea4e3ab659131"}
Sep 19 15:30:25.149035 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.149003    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"]
Sep 19 15:30:25.151831 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.151808    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:25.153881 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.153859    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-21d2a4-dockercfg-9ckhk\""
Sep 19 15:30:25.153881 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.153875    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-21d2a4\""
Sep 19 15:30:25.161708 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.161686    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"]
Sep 19 15:30:25.260215 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.260182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223e26a2-8d92-4b15-a565-ede51d9530d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm\" (UID: \"223e26a2-8d92-4b15-a565-ede51d9530d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:25.260444 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.260419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brdd\" (UniqueName: \"kubernetes.io/projected/223e26a2-8d92-4b15-a565-ede51d9530d0-kube-api-access-6brdd\") pod \"payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm\" (UID: \"223e26a2-8d92-4b15-a565-ede51d9530d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:25.361748 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.361720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brdd\" (UniqueName: \"kubernetes.io/projected/223e26a2-8d92-4b15-a565-ede51d9530d0-kube-api-access-6brdd\") pod \"payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm\" (UID: \"223e26a2-8d92-4b15-a565-ede51d9530d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:25.361879 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.361764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223e26a2-8d92-4b15-a565-ede51d9530d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm\" (UID: \"223e26a2-8d92-4b15-a565-ede51d9530d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:25.362553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.362528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223e26a2-8d92-4b15-a565-ede51d9530d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm\" (UID: \"223e26a2-8d92-4b15-a565-ede51d9530d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:25.373016 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.372992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brdd\" (UniqueName: \"kubernetes.io/projected/223e26a2-8d92-4b15-a565-ede51d9530d0-kube-api-access-6brdd\") pod \"payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm\" (UID: \"223e26a2-8d92-4b15-a565-ede51d9530d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:25.463517 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.463491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:25.735724 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:25.735614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"]
Sep 19 15:30:25.739104 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:25.739078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223e26a2_8d92_4b15_a565_ede51d9530d0.slice/crio-627cf0de40bf26381e3b29b44153cb5b58bb2cd169f8f8fd5a98d6beef469658 WatchSource:0}: Error finding container 627cf0de40bf26381e3b29b44153cb5b58bb2cd169f8f8fd5a98d6beef469658: Status 404 returned error can't find the container with id 627cf0de40bf26381e3b29b44153cb5b58bb2cd169f8f8fd5a98d6beef469658
Sep 19 15:30:26.222826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.222785    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"]
Sep 19 15:30:26.231410 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.231389    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:26.231547 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.231416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm" event={"ID":"223e26a2-8d92-4b15-a565-ede51d9530d0","Type":"ContainerStarted","Data":"d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e"}
Sep 19 15:30:26.231547 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.231443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm" event={"ID":"223e26a2-8d92-4b15-a565-ede51d9530d0","Type":"ContainerStarted","Data":"627cf0de40bf26381e3b29b44153cb5b58bb2cd169f8f8fd5a98d6beef469658"}
Sep 19 15:30:26.231547 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.231518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:26.234176 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.233847    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-15949d-dockercfg-bn4c4\""
Sep 19 15:30:26.234176 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.233881    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-15949d\""
Sep 19 15:30:26.235076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.235050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"]
Sep 19 15:30:26.246136 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.246094    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm" podStartSLOduration=1.246081343 podStartE2EDuration="1.246081343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:26.243255018 +0000 UTC m=+1088.136818656" watchObservedRunningTime="2026-09-19 15:30:26.246081343 +0000 UTC m=+1088.139644982"
Sep 19 15:30:26.271301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.271273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp\" (UID: \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:26.271441 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.271369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvgtg\" (UniqueName: \"kubernetes.io/projected/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-kube-api-access-bvgtg\") pod \"payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp\" (UID: \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:26.271441 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.271375    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"]
Sep 19 15:30:26.276210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.276181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:26.285900 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.285877    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"]
Sep 19 15:30:26.372914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.372883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz\" (UID: \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:26.373077 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.372933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvgtg\" (UniqueName: \"kubernetes.io/projected/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-kube-api-access-bvgtg\") pod \"payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp\" (UID: \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:26.373077 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.373058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4pzl\" (UniqueName: \"kubernetes.io/projected/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-kube-api-access-q4pzl\") pod \"payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz\" (UID: \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:26.373189 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.373152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp\" (UID: \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:26.374067 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.374039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp\" (UID: \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:26.380323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.380298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvgtg\" (UniqueName: \"kubernetes.io/projected/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-kube-api-access-bvgtg\") pod \"payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp\" (UID: \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:26.474174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.474104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4pzl\" (UniqueName: \"kubernetes.io/projected/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-kube-api-access-q4pzl\") pod \"payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz\" (UID: \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:26.474301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.474171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz\" (UID: \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:26.474825 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.474806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz\" (UID: \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:26.481373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.481353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4pzl\" (UniqueName: \"kubernetes.io/projected/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-kube-api-access-q4pzl\") pod \"payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz\" (UID: \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:26.546616 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.546571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:26.588437 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:26.588411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:27.009669 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.009646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"]
Sep 19 15:30:27.011923 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:27.011887    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ae52b9_f2a5_404f_a42d_25e2b0cd4ddb.slice/crio-81053f7bed862972f0fa61d067cbae5d0f455f03adbdd2c0cc0dd6fe95cb1899 WatchSource:0}: Error finding container 81053f7bed862972f0fa61d067cbae5d0f455f03adbdd2c0cc0dd6fe95cb1899: Status 404 returned error can't find the container with id 81053f7bed862972f0fa61d067cbae5d0f455f03adbdd2c0cc0dd6fe95cb1899
Sep 19 15:30:27.012442 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.012421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"]
Sep 19 15:30:27.014993 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:27.014972    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb5b58b_de14_47ab_a2cd_9aa963f7e668.slice/crio-5e9d93481d6fa99601bae267d05704cea27c303cdbc1fb19fc80548e6d39e390 WatchSource:0}: Error finding container 5e9d93481d6fa99601bae267d05704cea27c303cdbc1fb19fc80548e6d39e390: Status 404 returned error can't find the container with id 5e9d93481d6fa99601bae267d05704cea27c303cdbc1fb19fc80548e6d39e390
Sep 19 15:30:27.229187 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.229154    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"]
Sep 19 15:30:27.232545 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.232516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp" event={"ID":"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb","Type":"ContainerStarted","Data":"c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd"}
Sep 19 15:30:27.232703 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.232560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp" event={"ID":"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb","Type":"ContainerStarted","Data":"81053f7bed862972f0fa61d067cbae5d0f455f03adbdd2c0cc0dd6fe95cb1899"}
Sep 19 15:30:27.232703 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.232666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:27.232857 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.232797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:27.234196 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.234172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz" event={"ID":"4cb5b58b-de14-47ab-a2cd-9aa963f7e668","Type":"ContainerStarted","Data":"aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90"}
Sep 19 15:30:27.234301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.234207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz" event={"ID":"4cb5b58b-de14-47ab-a2cd-9aa963f7e668","Type":"ContainerStarted","Data":"5e9d93481d6fa99601bae267d05704cea27c303cdbc1fb19fc80548e6d39e390"}
Sep 19 15:30:27.234301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.234251    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:27.234795 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.234772    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-33737b\""
Sep 19 15:30:27.234894 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.234784    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-33737b-dockercfg-9b4nh\""
Sep 19 15:30:27.235722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.235703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9301-3a3727" event={"ID":"e337557f-07ff-469e-adb1-49a4573d963f","Type":"ContainerStarted","Data":"8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b"}
Sep 19 15:30:27.242913 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.242865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"]
Sep 19 15:30:27.255679 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.255631    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp" podStartSLOduration=1.25559674 podStartE2EDuration="1.25559674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:27.254305479 +0000 UTC m=+1089.147869120" watchObservedRunningTime="2026-09-19 15:30:27.25559674 +0000 UTC m=+1089.149160381"
Sep 19 15:30:27.262861 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.262810    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"]
Sep 19 15:30:27.266424 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.266408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:27.272975 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.272955    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"]
Sep 19 15:30:27.282918 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.282409    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9301-3a3727" podStartSLOduration=2.035098634 podStartE2EDuration="6.282392776s" totalImagesPullingTime="4.247294142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:27.27989441 +0000 UTC m=+1089.173458046" watchObservedRunningTime="2026-09-19 15:30:27.282392776 +0000 UTC m=+1089.175956417"
Sep 19 15:30:27.282918 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.282623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d\" (UID: \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:27.283468 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.283441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrc4n\" (UniqueName: \"kubernetes.io/projected/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-kube-api-access-hrc4n\") pod \"payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d\" (UID: \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:27.294453 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.294418    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz" podStartSLOduration=1.294406684 podStartE2EDuration="1.294406684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:27.293198037 +0000 UTC m=+1089.186761676" watchObservedRunningTime="2026-09-19 15:30:27.294406684 +0000 UTC m=+1089.187970365"
Sep 19 15:30:27.384511 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.384483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d0f7d5-2e94-4aaf-be23-85251088c4b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx\" (UID: \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:27.384647 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.384543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d\" (UID: \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:27.384647 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.384628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktzv\" (UniqueName: \"kubernetes.io/projected/79d0f7d5-2e94-4aaf-be23-85251088c4b4-kube-api-access-7ktzv\") pod \"payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx\" (UID: \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:27.384758 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.384673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrc4n\" (UniqueName: \"kubernetes.io/projected/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-kube-api-access-hrc4n\") pod \"payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d\" (UID: \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:27.385248 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.385230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d\" (UID: \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:27.391871 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.391849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrc4n\" (UniqueName: \"kubernetes.io/projected/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-kube-api-access-hrc4n\") pod \"payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d\" (UID: \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:27.485401 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.485374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d0f7d5-2e94-4aaf-be23-85251088c4b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx\" (UID: \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:27.485494 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.485470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktzv\" (UniqueName: \"kubernetes.io/projected/79d0f7d5-2e94-4aaf-be23-85251088c4b4-kube-api-access-7ktzv\") pod \"payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx\" (UID: \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:27.486213 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.486180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d0f7d5-2e94-4aaf-be23-85251088c4b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx\" (UID: \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:27.494113 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.494089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktzv\" (UniqueName: \"kubernetes.io/projected/79d0f7d5-2e94-4aaf-be23-85251088c4b4-kube-api-access-7ktzv\") pod \"payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx\" (UID: \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:27.545789 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.545732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:27.579153 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.579130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:27.827864 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.827839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"]
Sep 19 15:30:27.829785 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:27.829759    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f3601e_4fd4_4936_ba9f_f0a25db7d5f6.slice/crio-86ca838f6077c7007c186b8de6a06f40f208e7b1273ad8a3ec99f459239230f0 WatchSource:0}: Error finding container 86ca838f6077c7007c186b8de6a06f40f208e7b1273ad8a3ec99f459239230f0: Status 404 returned error can't find the container with id 86ca838f6077c7007c186b8de6a06f40f208e7b1273ad8a3ec99f459239230f0
Sep 19 15:30:27.858271 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.858246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"]
Sep 19 15:30:27.866824 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:27.866798    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d0f7d5_2e94_4aaf_be23_85251088c4b4.slice/crio-e16fe9d9977edef9503dc897137ccfa213eaa03cc6b25e23be4c5af63ad8cf21 WatchSource:0}: Error finding container e16fe9d9977edef9503dc897137ccfa213eaa03cc6b25e23be4c5af63ad8cf21: Status 404 returned error can't find the container with id e16fe9d9977edef9503dc897137ccfa213eaa03cc6b25e23be4c5af63ad8cf21
Sep 19 15:30:27.901960 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.901928    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9301-3a3727"]
Sep 19 15:30:27.905423 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:27.904899    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9301-3a3727"]
Sep 19 15:30:28.243566 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:28.243531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d" event={"ID":"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6","Type":"ContainerStarted","Data":"d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d"}
Sep 19 15:30:28.244122 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:28.244082    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:28.244217 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:28.244120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d" event={"ID":"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6","Type":"ContainerStarted","Data":"86ca838f6077c7007c186b8de6a06f40f208e7b1273ad8a3ec99f459239230f0"}
Sep 19 15:30:28.245455 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:28.245423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx" event={"ID":"79d0f7d5-2e94-4aaf-be23-85251088c4b4","Type":"ContainerStarted","Data":"e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a"}
Sep 19 15:30:28.245574 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:28.245477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx" event={"ID":"79d0f7d5-2e94-4aaf-be23-85251088c4b4","Type":"ContainerStarted","Data":"e16fe9d9977edef9503dc897137ccfa213eaa03cc6b25e23be4c5af63ad8cf21"}
Sep 19 15:30:28.245764 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:30:28.245726    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9301-3a3727" container="test-curl-9301-3a3727"
Sep 19 15:30:28.258020 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:28.257977    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d" podStartSLOduration=1.25795969 podStartE2EDuration="1.25795969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:28.257024311 +0000 UTC m=+1090.150587953" watchObservedRunningTime="2026-09-19 15:30:28.25795969 +0000 UTC m=+1090.151523332"
Sep 19 15:30:28.272621 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:28.272392    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx" podStartSLOduration=1.27237634 podStartE2EDuration="1.27237634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:28.270802703 +0000 UTC m=+1090.164366344" watchObservedRunningTime="2026-09-19 15:30:28.27237634 +0000 UTC m=+1090.165939979"
Sep 19 15:30:29.250725 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:29.250683    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9301-3a3727" podUID="e337557f-07ff-469e-adb1-49a4573d963f" containerName="test-curl-9301-3a3727" containerID="cri-o://8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b" gracePeriod=2
Sep 19 15:30:29.251189 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:29.250856    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:31.415760 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:31.415741    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9301-3a3727"
Sep 19 15:30:31.524815 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:31.524750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl5bv\" (UniqueName: \"kubernetes.io/projected/e337557f-07ff-469e-adb1-49a4573d963f-kube-api-access-nl5bv\") pod \"e337557f-07ff-469e-adb1-49a4573d963f\" (UID: \"e337557f-07ff-469e-adb1-49a4573d963f\") "
Sep 19 15:30:31.526939 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:31.526912    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e337557f-07ff-469e-adb1-49a4573d963f-kube-api-access-nl5bv" (OuterVolumeSpecName: "kube-api-access-nl5bv") pod "e337557f-07ff-469e-adb1-49a4573d963f" (UID: "e337557f-07ff-469e-adb1-49a4573d963f"). InnerVolumeSpecName "kube-api-access-nl5bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:30:31.625906 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:31.625882    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl5bv\" (UniqueName: \"kubernetes.io/projected/e337557f-07ff-469e-adb1-49a4573d963f-kube-api-access-nl5bv\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:30:32.170918 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:32.170889    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e337557f-07ff-469e-adb1-49a4573d963f" path="/var/lib/kubelet/pods/e337557f-07ff-469e-adb1-49a4573d963f/volumes"
Sep 19 15:30:32.266927 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:32.266893    2579 generic.go:356] "Generic (PLEG): container finished" podID="e337557f-07ff-469e-adb1-49a4573d963f" containerID="8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b" exitCode=137
Sep 19 15:30:32.267025 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:32.266934    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9301-3a3727"
Sep 19 15:30:32.267025 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:32.266975    2579 scope.go:117] "RemoveContainer" containerID="8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b"
Sep 19 15:30:32.274847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:32.274830    2579 scope.go:117] "RemoveContainer" containerID="8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b"
Sep 19 15:30:32.275101 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:30:32.275085    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b\": container with ID starting with 8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b not found: ID does not exist" containerID="8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b"
Sep 19 15:30:32.275150 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:32.275107    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b"} err="failed to get container status \"8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b\": rpc error: code = NotFound desc = could not find container \"8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b\": container with ID starting with 8f802fa052fc79a9a7c8d03a336489e93c79533d6e695888e0b6e57e6dc0455b not found: ID does not exist"
Sep 19 15:30:35.297182 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.297153    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"]
Sep 19 15:30:35.297669 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.297651    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e337557f-07ff-469e-adb1-49a4573d963f" containerName="test-curl-9301-3a3727"
Sep 19 15:30:35.297765 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.297671    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e337557f-07ff-469e-adb1-49a4573d963f" containerName="test-curl-9301-3a3727"
Sep 19 15:30:35.297817 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.297792    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e337557f-07ff-469e-adb1-49a4573d963f" containerName="test-curl-9301-3a3727"
Sep 19 15:30:35.302210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.302190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:35.304199 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.304170    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e86b6784-dockercfg-tctqd\""
Sep 19 15:30:35.304342 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.304322    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e86b6784\""
Sep 19 15:30:35.309548 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.309525    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"]
Sep 19 15:30:35.331402 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.331377    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"]
Sep 19 15:30:35.334512 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.334497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:35.343695 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.343673    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"]
Sep 19 15:30:35.354248 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.354225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx\" (UID: \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:35.354352 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.354278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntgc\" (UniqueName: \"kubernetes.io/projected/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-kube-api-access-tntgc\") pod \"payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx\" (UID: \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:35.455103 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.455082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx\" (UID: \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:35.455201 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.455119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd74b\" (UniqueName: \"kubernetes.io/projected/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-kube-api-access-rd74b\") pod \"payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk\" (UID: \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\") " pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:35.455201 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.455144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-plugins-config-volume\") pod \"payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk\" (UID: \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\") " pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:35.455201 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.455169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntgc\" (UniqueName: \"kubernetes.io/projected/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-kube-api-access-tntgc\") pod \"payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx\" (UID: \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:35.455813 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.455792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx\" (UID: \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:35.463074 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.463058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntgc\" (UniqueName: \"kubernetes.io/projected/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-kube-api-access-tntgc\") pod \"payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx\" (UID: \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:35.556154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.556095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-plugins-config-volume\") pod \"payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk\" (UID: \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\") " pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:35.556249 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.556170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd74b\" (UniqueName: \"kubernetes.io/projected/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-kube-api-access-rd74b\") pod \"payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk\" (UID: \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\") " pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:35.556754 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.556738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-plugins-config-volume\") pod \"payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk\" (UID: \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\") " pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:35.563194 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.563165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd74b\" (UniqueName: \"kubernetes.io/projected/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-kube-api-access-rd74b\") pod \"payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk\" (UID: \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\") " pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:35.613182 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.613160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:35.644435 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.644411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:35.884634 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.884591    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"]
Sep 19 15:30:35.886171 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:35.886135    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e4ef64_93cc_4cc7_8dba_ea381b369c4c.slice/crio-21c2ae6ab25254ef9b8c7f4b0a9f35b5a8dbb4f44ee4af8c0242c351c6f92ab1 WatchSource:0}: Error finding container 21c2ae6ab25254ef9b8c7f4b0a9f35b5a8dbb4f44ee4af8c0242c351c6f92ab1: Status 404 returned error can't find the container with id 21c2ae6ab25254ef9b8c7f4b0a9f35b5a8dbb4f44ee4af8c0242c351c6f92ab1
Sep 19 15:30:35.923988 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:35.923967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"]
Sep 19 15:30:35.926367 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:35.926341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e2fd70_db9f_4e93_bf1c_6d60d224fef4.slice/crio-0ee4b10f612170eb30371296cf39ae81fda2b5b5d28990e613c04dea06b904fe WatchSource:0}: Error finding container 0ee4b10f612170eb30371296cf39ae81fda2b5b5d28990e613c04dea06b904fe: Status 404 returned error can't find the container with id 0ee4b10f612170eb30371296cf39ae81fda2b5b5d28990e613c04dea06b904fe
Sep 19 15:30:36.288147 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:36.288115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx" event={"ID":"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c","Type":"ContainerStarted","Data":"63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7"}
Sep 19 15:30:36.288318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:36.288175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:36.288318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:36.288196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx" event={"ID":"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c","Type":"ContainerStarted","Data":"21c2ae6ab25254ef9b8c7f4b0a9f35b5a8dbb4f44ee4af8c0242c351c6f92ab1"}
Sep 19 15:30:36.290225 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:36.290201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk" event={"ID":"66e2fd70-db9f-4e93-bf1c-6d60d224fef4","Type":"ContainerStarted","Data":"a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e"}
Sep 19 15:30:36.290368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:36.290240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk" event={"ID":"66e2fd70-db9f-4e93-bf1c-6d60d224fef4","Type":"ContainerStarted","Data":"0ee4b10f612170eb30371296cf39ae81fda2b5b5d28990e613c04dea06b904fe"}
Sep 19 15:30:36.290368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:36.290311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:36.304622 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:36.304562    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx" podStartSLOduration=1.304547956 podStartE2EDuration="1.304547956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:36.301835954 +0000 UTC m=+1098.195399593" watchObservedRunningTime="2026-09-19 15:30:36.304547956 +0000 UTC m=+1098.198111644"
Sep 19 15:30:36.316297 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:36.316256    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk" podStartSLOduration=1.316245885 podStartE2EDuration="1.316245885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:36.315236172 +0000 UTC m=+1098.208799814" watchObservedRunningTime="2026-09-19 15:30:36.316245885 +0000 UTC m=+1098.209809535"
Sep 19 15:30:37.238119 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:37.238091    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:30:38.247671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:38.247644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:30:38.248128 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:38.247719    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:30:39.252739 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:39.252708    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:30:40.259109 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:40.259081    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:30:43.603922 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.603890    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"]
Sep 19 15:30:43.610441 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.610421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:43.612547 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.612526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e93f57\""
Sep 19 15:30:43.612672 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.612569    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e93f57-dockercfg-4tcnk\""
Sep 19 15:30:43.617762 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.617737    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"]
Sep 19 15:30:43.642835 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.642807    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"]
Sep 19 15:30:43.646143 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.646127    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:43.657699 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.657677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"]
Sep 19 15:30:43.726446 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.726414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9t7\" (UniqueName: \"kubernetes.io/projected/4c521541-0eb0-4bc8-9a6d-da6832616a0e-kube-api-access-ks9t7\") pod \"payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm\" (UID: \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:43.726626 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.726458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c521541-0eb0-4bc8-9a6d-da6832616a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm\" (UID: \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:43.726626 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.726569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw992\" (UniqueName: \"kubernetes.io/projected/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-kube-api-access-rw992\") pod \"payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t\" (UID: \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:43.726719 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.726635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t\" (UID: \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:43.827148 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.827108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw992\" (UniqueName: \"kubernetes.io/projected/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-kube-api-access-rw992\") pod \"payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t\" (UID: \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:43.827325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.827186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t\" (UID: \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:43.827325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.827237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9t7\" (UniqueName: \"kubernetes.io/projected/4c521541-0eb0-4bc8-9a6d-da6832616a0e-kube-api-access-ks9t7\") pod \"payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm\" (UID: \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:43.827325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.827290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c521541-0eb0-4bc8-9a6d-da6832616a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm\" (UID: \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:43.827949 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.827930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t\" (UID: \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:43.828045 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.828028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c521541-0eb0-4bc8-9a6d-da6832616a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm\" (UID: \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:43.838427 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.835752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9t7\" (UniqueName: \"kubernetes.io/projected/4c521541-0eb0-4bc8-9a6d-da6832616a0e-kube-api-access-ks9t7\") pod \"payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm\" (UID: \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:43.838427 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.835806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw992\" (UniqueName: \"kubernetes.io/projected/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-kube-api-access-rw992\") pod \"payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t\" (UID: \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:43.922735 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.922647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:43.956085 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:43.956053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:44.195569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.195540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"]
Sep 19 15:30:44.197226 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:44.197197    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c521541_0eb0_4bc8_9a6d_da6832616a0e.slice/crio-d2922abc3510f9b7faf6619c2c3d3acbf1ce9700876d2a9ab29f295980a406ff WatchSource:0}: Error finding container d2922abc3510f9b7faf6619c2c3d3acbf1ce9700876d2a9ab29f295980a406ff: Status 404 returned error can't find the container with id d2922abc3510f9b7faf6619c2c3d3acbf1ce9700876d2a9ab29f295980a406ff
Sep 19 15:30:44.234446 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.234420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"]
Sep 19 15:30:44.235702 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:30:44.235675    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1a75b2_7ff2_43f4_86e8_aea0d0933acb.slice/crio-eb0893cdfc5a82160b89b2be2c31d04797f7c211c8e30b3ef2e5fa6a45dadd91 WatchSource:0}: Error finding container eb0893cdfc5a82160b89b2be2c31d04797f7c211c8e30b3ef2e5fa6a45dadd91: Status 404 returned error can't find the container with id eb0893cdfc5a82160b89b2be2c31d04797f7c211c8e30b3ef2e5fa6a45dadd91
Sep 19 15:30:44.331801 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.331766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t" event={"ID":"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb","Type":"ContainerStarted","Data":"aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1"}
Sep 19 15:30:44.331960 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.331835    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:44.331960 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.331857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t" event={"ID":"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb","Type":"ContainerStarted","Data":"eb0893cdfc5a82160b89b2be2c31d04797f7c211c8e30b3ef2e5fa6a45dadd91"}
Sep 19 15:30:44.333474 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.333451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm" event={"ID":"4c521541-0eb0-4bc8-9a6d-da6832616a0e","Type":"ContainerStarted","Data":"60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f"}
Sep 19 15:30:44.333576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.333487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm" event={"ID":"4c521541-0eb0-4bc8-9a6d-da6832616a0e","Type":"ContainerStarted","Data":"d2922abc3510f9b7faf6619c2c3d3acbf1ce9700876d2a9ab29f295980a406ff"}
Sep 19 15:30:44.333576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.333519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:30:44.349947 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.349897    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t" podStartSLOduration=1.349877191 podStartE2EDuration="1.349877191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:44.347224683 +0000 UTC m=+1106.240788327" watchObservedRunningTime="2026-09-19 15:30:44.349877191 +0000 UTC m=+1106.243440832"
Sep 19 15:30:44.363304 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:44.363260    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm" podStartSLOduration=1.363241626 podStartE2EDuration="1.363241626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:30:44.361791691 +0000 UTC m=+1106.255355328" watchObservedRunningTime="2026-09-19 15:30:44.363241626 +0000 UTC m=+1106.256805268"
Sep 19 15:30:47.297994 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:47.297954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:30:47.298340 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:47.298031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:30:55.340786 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:55.340757    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:30:55.341177 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:30:55.340827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:31:02.091352 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.091318    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"]
Sep 19 15:31:02.098008 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.097993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"
Sep 19 15:31:02.101175 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.101024    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 15:31:02.101175 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.101044    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7znsl\""
Sep 19 15:31:02.103127 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.103103    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"]
Sep 19 15:31:02.277208 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.277165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d2b8b7-f13f-4d4c-b318-9b86fc7697ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq\" (UID: \"41d2b8b7-f13f-4d4c-b318-9b86fc7697ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"
Sep 19 15:31:02.378077 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.378013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d2b8b7-f13f-4d4c-b318-9b86fc7697ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq\" (UID: \"41d2b8b7-f13f-4d4c-b318-9b86fc7697ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"
Sep 19 15:31:02.378559 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.378541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d2b8b7-f13f-4d4c-b318-9b86fc7697ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq\" (UID: \"41d2b8b7-f13f-4d4c-b318-9b86fc7697ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"
Sep 19 15:31:02.410167 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.410147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"
Sep 19 15:31:02.676791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:02.676763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"]
Sep 19 15:31:02.678045 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:31:02.678017    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d2b8b7_f13f_4d4c_b318_9b86fc7697ce.slice/crio-563e8b1459e37b6e0677521142df4e7bd18c40fba404b6152a9ee8d9b1921d06 WatchSource:0}: Error finding container 563e8b1459e37b6e0677521142df4e7bd18c40fba404b6152a9ee8d9b1921d06: Status 404 returned error can't find the container with id 563e8b1459e37b6e0677521142df4e7bd18c40fba404b6152a9ee8d9b1921d06
Sep 19 15:31:03.426714 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:03.426679    2579 generic.go:356] "Generic (PLEG): container finished" podID="41d2b8b7-f13f-4d4c-b318-9b86fc7697ce" containerID="742ed2502ca0b745978c9333e672212b0332d7c54ff44998f08b201608ddee0e" exitCode=0
Sep 19 15:31:03.427085 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:03.426732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq" event={"ID":"41d2b8b7-f13f-4d4c-b318-9b86fc7697ce","Type":"ContainerDied","Data":"742ed2502ca0b745978c9333e672212b0332d7c54ff44998f08b201608ddee0e"}
Sep 19 15:31:03.427085 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:03.426782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq" event={"ID":"41d2b8b7-f13f-4d4c-b318-9b86fc7697ce","Type":"ContainerStarted","Data":"563e8b1459e37b6e0677521142df4e7bd18c40fba404b6152a9ee8d9b1921d06"}
Sep 19 15:31:04.554235 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:04.554216    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"
Sep 19 15:31:04.595907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:04.595882    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d2b8b7-f13f-4d4c-b318-9b86fc7697ce-maas-api-service-ca\") pod \"41d2b8b7-f13f-4d4c-b318-9b86fc7697ce\" (UID: \"41d2b8b7-f13f-4d4c-b318-9b86fc7697ce\") "
Sep 19 15:31:04.596196 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:04.596174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d2b8b7-f13f-4d4c-b318-9b86fc7697ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41d2b8b7-f13f-4d4c-b318-9b86fc7697ce" (UID: "41d2b8b7-f13f-4d4c-b318-9b86fc7697ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:31:04.696535 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:04.696466    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d2b8b7-f13f-4d4c-b318-9b86fc7697ce-maas-api-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:05.436503 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:05.436477    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"
Sep 19 15:31:05.436690 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:05.436471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq" event={"ID":"41d2b8b7-f13f-4d4c-b318-9b86fc7697ce","Type":"ContainerDied","Data":"563e8b1459e37b6e0677521142df4e7bd18c40fba404b6152a9ee8d9b1921d06"}
Sep 19 15:31:05.436690 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:05.436596    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563e8b1459e37b6e0677521142df4e7bd18c40fba404b6152a9ee8d9b1921d06"
Sep 19 15:31:06.835924 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.835891    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"]
Sep 19 15:31:06.836431 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.836411    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d2b8b7-f13f-4d4c-b318-9b86fc7697ce" containerName="revoke-keys"
Sep 19 15:31:06.836513 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.836434    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d2b8b7-f13f-4d4c-b318-9b86fc7697ce" containerName="revoke-keys"
Sep 19 15:31:06.836564 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.836547    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d2b8b7-f13f-4d4c-b318-9b86fc7697ce" containerName="revoke-keys"
Sep 19 15:31:06.839411 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.839390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:06.842950 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.842932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-33737b\"/\"openshift-service-ca.crt\""
Sep 19 15:31:06.843905 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.843884    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-33737b\"/\"simulator-w3-33737b-kserve-self-signed-certs\""
Sep 19 15:31:06.844018 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.843942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-33737b\"/\"kube-root-ca.crt\""
Sep 19 15:31:06.844018 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.843949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-33737b\"/\"default-dockercfg-4tlbw\""
Sep 19 15:31:06.849556 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.849526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"]
Sep 19 15:31:06.915873 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.915841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tmp-dir\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:06.916017 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.915879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-model-cache\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:06.916017 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.915908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-home\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:06.916017 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.915980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tls-certs\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:06.916017 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.916006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k29v2\" (UniqueName: \"kubernetes.io/projected/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-kube-api-access-k29v2\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:06.916160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:06.916038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-dshm\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.016957 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.016930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-dshm\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.017114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.016994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tmp-dir\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.017114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.017015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-model-cache\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.017114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.017048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-home\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.017114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.017068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tls-certs\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.017114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.017083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k29v2\" (UniqueName: \"kubernetes.io/projected/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-kube-api-access-k29v2\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.017458 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.017436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tmp-dir\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.017458 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.017447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-model-cache\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.017555 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.017510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-home\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.019228 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.019213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-dshm\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.019558 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.019542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tls-certs\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.024853 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.024833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k29v2\" (UniqueName: \"kubernetes.io/projected/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-kube-api-access-k29v2\") pod \"simulator-w3-33737b-kserve-5ffdc5855f-c6fh6\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") " pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.149697 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.149597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:07.424445 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.418213    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"]
Sep 19 15:31:07.446142 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:07.446114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6" event={"ID":"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38","Type":"ContainerStarted","Data":"737a63c215489c08f0f43dd392d82ecbdfce4958be0d0f1be6530e1b682699c5"}
Sep 19 15:31:08.453434 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:08.453403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6" event={"ID":"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38","Type":"ContainerStarted","Data":"4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277"}
Sep 19 15:31:08.468584 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:08.468539    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6" podStartSLOduration=2.468524477 podStartE2EDuration="2.468524477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:31:08.467099981 +0000 UTC m=+1130.360663620" watchObservedRunningTime="2026-09-19 15:31:08.468524477 +0000 UTC m=+1130.362088117"
Sep 19 15:31:13.575786 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.575751    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"]
Sep 19 15:31:13.576470 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.576042    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk" podUID="66e2fd70-db9f-4e93-bf1c-6d60d224fef4" containerName="payload-processing" containerID="cri-o://a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e" gracePeriod=30
Sep 19 15:31:13.582783 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.582744    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"]
Sep 19 15:31:13.583008 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.582979    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx" podUID="e7e4ef64-93cc-4cc7-8dba-ea381b369c4c" containerName="payload-pre-processing" containerID="cri-o://63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7" gracePeriod=30
Sep 19 15:31:13.720213 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.720167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"]
Sep 19 15:31:13.721729 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.721701    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e86b6784-tq7pq"]
Sep 19 15:31:13.849752 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.849728    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:31:13.851994 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.851973    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:31:13.876737 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.876713    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-plugins-config-volume\") pod \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\" (UID: \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\") "
Sep 19 15:31:13.876891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.876750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-plugins-config-volume\") pod \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\" (UID: \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\") "
Sep 19 15:31:13.876891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.876826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tntgc\" (UniqueName: \"kubernetes.io/projected/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-kube-api-access-tntgc\") pod \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\" (UID: \"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c\") "
Sep 19 15:31:13.876891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.876849    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd74b\" (UniqueName: \"kubernetes.io/projected/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-kube-api-access-rd74b\") pod \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\" (UID: \"66e2fd70-db9f-4e93-bf1c-6d60d224fef4\") "
Sep 19 15:31:13.877097 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.877070    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7e4ef64-93cc-4cc7-8dba-ea381b369c4c" (UID: "e7e4ef64-93cc-4cc7-8dba-ea381b369c4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:31:13.877195 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.877172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66e2fd70-db9f-4e93-bf1c-6d60d224fef4" (UID: "66e2fd70-db9f-4e93-bf1c-6d60d224fef4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:31:13.878992 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.878967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-kube-api-access-rd74b" (OuterVolumeSpecName: "kube-api-access-rd74b") pod "66e2fd70-db9f-4e93-bf1c-6d60d224fef4" (UID: "66e2fd70-db9f-4e93-bf1c-6d60d224fef4"). InnerVolumeSpecName "kube-api-access-rd74b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:31:13.879103 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.879065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-kube-api-access-tntgc" (OuterVolumeSpecName: "kube-api-access-tntgc") pod "e7e4ef64-93cc-4cc7-8dba-ea381b369c4c" (UID: "e7e4ef64-93cc-4cc7-8dba-ea381b369c4c"). InnerVolumeSpecName "kube-api-access-tntgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:31:13.978404 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.978376    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntgc\" (UniqueName: \"kubernetes.io/projected/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-kube-api-access-tntgc\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:13.978404 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.978400    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd74b\" (UniqueName: \"kubernetes.io/projected/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-kube-api-access-rd74b\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:13.978404 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.978410    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:13.978583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:13.978419    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2fd70-db9f-4e93-bf1c-6d60d224fef4-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:14.171454 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.171392    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d2b8b7-f13f-4d4c-b318-9b86fc7697ce" path="/var/lib/kubelet/pods/41d2b8b7-f13f-4d4c-b318-9b86fc7697ce/volumes"
Sep 19 15:31:14.482356 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.482324    2579 generic.go:356] "Generic (PLEG): container finished" podID="66e2fd70-db9f-4e93-bf1c-6d60d224fef4" containerID="a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e" exitCode=0
Sep 19 15:31:14.482535 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.482369    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"
Sep 19 15:31:14.482535 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.482451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk" event={"ID":"66e2fd70-db9f-4e93-bf1c-6d60d224fef4","Type":"ContainerDied","Data":"a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e"}
Sep 19 15:31:14.482535 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.482510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk" event={"ID":"66e2fd70-db9f-4e93-bf1c-6d60d224fef4","Type":"ContainerDied","Data":"0ee4b10f612170eb30371296cf39ae81fda2b5b5d28990e613c04dea06b904fe"}
Sep 19 15:31:14.482745 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.482541    2579 scope.go:117] "RemoveContainer" containerID="a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e"
Sep 19 15:31:14.484302 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.484158    2579 generic.go:356] "Generic (PLEG): container finished" podID="e7e4ef64-93cc-4cc7-8dba-ea381b369c4c" containerID="63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7" exitCode=0
Sep 19 15:31:14.484302 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.484204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx" event={"ID":"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c","Type":"ContainerDied","Data":"63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7"}
Sep 19 15:31:14.484302 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.484211    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"
Sep 19 15:31:14.484302 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.484252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx" event={"ID":"e7e4ef64-93cc-4cc7-8dba-ea381b369c4c","Type":"ContainerDied","Data":"21c2ae6ab25254ef9b8c7f4b0a9f35b5a8dbb4f44ee4af8c0242c351c6f92ab1"}
Sep 19 15:31:14.491876 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.491859    2579 scope.go:117] "RemoveContainer" containerID="a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e"
Sep 19 15:31:14.492137 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:31:14.492119    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e\": container with ID starting with a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e not found: ID does not exist" containerID="a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e"
Sep 19 15:31:14.492181 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.492146    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e"} err="failed to get container status \"a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e\": rpc error: code = NotFound desc = could not find container \"a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e\": container with ID starting with a3dec9970518428b1223057fd35cfb77477e0f387a4a14041000a0f9bd21fd2e not found: ID does not exist"
Sep 19 15:31:14.492181 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.492163    2579 scope.go:117] "RemoveContainer" containerID="63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7"
Sep 19 15:31:14.499491 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.499473    2579 scope.go:117] "RemoveContainer" containerID="63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7"
Sep 19 15:31:14.499782 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:31:14.499765    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7\": container with ID starting with 63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7 not found: ID does not exist" containerID="63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7"
Sep 19 15:31:14.499844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.499787    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7"} err="failed to get container status \"63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7\": rpc error: code = NotFound desc = could not find container \"63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7\": container with ID starting with 63262350fe7704393f5d5edb868208cb0f7d59aee2f436ba5d824dd2a65120c7 not found: ID does not exist"
Sep 19 15:31:14.503296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.503277    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"]
Sep 19 15:31:14.508889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.508871    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e86b6784-68f95d9f95-h5fkk"]
Sep 19 15:31:14.517961 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.517941    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"]
Sep 19 15:31:14.522292 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:14.522273    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e86b6784-54d7dc56db-l2dbx"]
Sep 19 15:31:16.170662 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:16.170632    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e2fd70-db9f-4e93-bf1c-6d60d224fef4" path="/var/lib/kubelet/pods/66e2fd70-db9f-4e93-bf1c-6d60d224fef4/volumes"
Sep 19 15:31:16.171008 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:16.170957    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e4ef64-93cc-4cc7-8dba-ea381b369c4c" path="/var/lib/kubelet/pods/e7e4ef64-93cc-4cc7-8dba-ea381b369c4c/volumes"
Sep 19 15:31:17.150016 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:17.149980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:17.150205 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:17.150039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:17.161945 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:17.161917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:17.512031 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:17.512005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:31:18.534778 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.534739    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"]
Sep 19 15:31:18.535278 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.535259    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e4ef64-93cc-4cc7-8dba-ea381b369c4c" containerName="payload-pre-processing"
Sep 19 15:31:18.535360 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.535280    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e4ef64-93cc-4cc7-8dba-ea381b369c4c" containerName="payload-pre-processing"
Sep 19 15:31:18.535360 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.535328    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e2fd70-db9f-4e93-bf1c-6d60d224fef4" containerName="payload-processing"
Sep 19 15:31:18.535360 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.535337    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e2fd70-db9f-4e93-bf1c-6d60d224fef4" containerName="payload-processing"
Sep 19 15:31:18.535507 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.535426    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e4ef64-93cc-4cc7-8dba-ea381b369c4c" containerName="payload-pre-processing"
Sep 19 15:31:18.535507 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.535438    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e2fd70-db9f-4e93-bf1c-6d60d224fef4" containerName="payload-processing"
Sep 19 15:31:18.539771 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.539750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:18.541752 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.541735    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a95f496f-dockercfg-p8b59\""
Sep 19 15:31:18.541839 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.541735    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a95f496f\""
Sep 19 15:31:18.547442 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.547421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"]
Sep 19 15:31:18.566884 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.566859    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"]
Sep 19 15:31:18.570135 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.570120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:18.578471 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.578450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"]
Sep 19 15:31:18.613386 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.613362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh\" (UID: \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:18.613486 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.613414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsgsj\" (UniqueName: \"kubernetes.io/projected/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-kube-api-access-rsgsj\") pod \"payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh\" (UID: \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:18.613486 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.613436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rss99\" (UniqueName: \"kubernetes.io/projected/4af21e20-a717-456b-8276-a204d01c57f0-kube-api-access-rss99\") pod \"payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw\" (UID: \"4af21e20-a717-456b-8276-a204d01c57f0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:18.613591 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.613529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af21e20-a717-456b-8276-a204d01c57f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw\" (UID: \"4af21e20-a717-456b-8276-a204d01c57f0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:18.714970 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.714944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsgsj\" (UniqueName: \"kubernetes.io/projected/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-kube-api-access-rsgsj\") pod \"payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh\" (UID: \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:18.715082 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.714983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rss99\" (UniqueName: \"kubernetes.io/projected/4af21e20-a717-456b-8276-a204d01c57f0-kube-api-access-rss99\") pod \"payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw\" (UID: \"4af21e20-a717-456b-8276-a204d01c57f0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:18.715082 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.715023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af21e20-a717-456b-8276-a204d01c57f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw\" (UID: \"4af21e20-a717-456b-8276-a204d01c57f0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:18.715194 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.715081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh\" (UID: \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:18.715799 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.715780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af21e20-a717-456b-8276-a204d01c57f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw\" (UID: \"4af21e20-a717-456b-8276-a204d01c57f0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:18.715835 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.715778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh\" (UID: \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:18.722572 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.722544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rss99\" (UniqueName: \"kubernetes.io/projected/4af21e20-a717-456b-8276-a204d01c57f0-kube-api-access-rss99\") pod \"payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw\" (UID: \"4af21e20-a717-456b-8276-a204d01c57f0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:18.722572 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.722567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsgsj\" (UniqueName: \"kubernetes.io/projected/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-kube-api-access-rsgsj\") pod \"payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh\" (UID: \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:18.850891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.850837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:18.880379 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:18.880347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:19.124677 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.124649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"]
Sep 19 15:31:19.126355 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:31:19.126323    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af21e20_a717_456b_8276_a204d01c57f0.slice/crio-81bb3a032822a240530619a3487b9f54c4f2ef0b515f020eb9e79764e7d647b4 WatchSource:0}: Error finding container 81bb3a032822a240530619a3487b9f54c4f2ef0b515f020eb9e79764e7d647b4: Status 404 returned error can't find the container with id 81bb3a032822a240530619a3487b9f54c4f2ef0b515f020eb9e79764e7d647b4
Sep 19 15:31:19.161322 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.161228    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"]
Sep 19 15:31:19.163715 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:31:19.163693    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eaa0a3_cbdc_407a_8c31_29e7197ec0cd.slice/crio-b5413a4237a482c60a538d74141628f0f1999bca66a0738c3b079d244c7bfe7e WatchSource:0}: Error finding container b5413a4237a482c60a538d74141628f0f1999bca66a0738c3b079d244c7bfe7e: Status 404 returned error can't find the container with id b5413a4237a482c60a538d74141628f0f1999bca66a0738c3b079d244c7bfe7e
Sep 19 15:31:19.513129 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.513095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw" event={"ID":"4af21e20-a717-456b-8276-a204d01c57f0","Type":"ContainerStarted","Data":"af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649"}
Sep 19 15:31:19.513294 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.513161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:19.513294 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.513184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw" event={"ID":"4af21e20-a717-456b-8276-a204d01c57f0","Type":"ContainerStarted","Data":"81bb3a032822a240530619a3487b9f54c4f2ef0b515f020eb9e79764e7d647b4"}
Sep 19 15:31:19.514706 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.514681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh" event={"ID":"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd","Type":"ContainerStarted","Data":"2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7"}
Sep 19 15:31:19.514828 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.514717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh" event={"ID":"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd","Type":"ContainerStarted","Data":"b5413a4237a482c60a538d74141628f0f1999bca66a0738c3b079d244c7bfe7e"}
Sep 19 15:31:19.514828 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.514812    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:19.527390 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.527349    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw" podStartSLOduration=1.5273352820000001 podStartE2EDuration="1.527335282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:31:19.525746589 +0000 UTC m=+1141.419310228" watchObservedRunningTime="2026-09-19 15:31:19.527335282 +0000 UTC m=+1141.420898921"
Sep 19 15:31:19.541324 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:19.541284    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh" podStartSLOduration=1.5412715590000001 podStartE2EDuration="1.541271559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:31:19.540232363 +0000 UTC m=+1141.433796014" watchObservedRunningTime="2026-09-19 15:31:19.541271559 +0000 UTC m=+1141.434835198"
Sep 19 15:31:30.521118 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:30.521044    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:30.521665 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:30.521120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:47.140460 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.140426    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"]
Sep 19 15:31:47.144105 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.144083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"
Sep 19 15:31:47.146034 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.146013    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 15:31:47.146157 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.146115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7znsl\""
Sep 19 15:31:47.150073 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.150048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"]
Sep 19 15:31:47.259724 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.259689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/239ec181-cbcc-4f01-aad7-af836e2224a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks\" (UID: \"239ec181-cbcc-4f01-aad7-af836e2224a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"
Sep 19 15:31:47.360994 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.360964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/239ec181-cbcc-4f01-aad7-af836e2224a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks\" (UID: \"239ec181-cbcc-4f01-aad7-af836e2224a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"
Sep 19 15:31:47.361563 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.361543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/239ec181-cbcc-4f01-aad7-af836e2224a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks\" (UID: \"239ec181-cbcc-4f01-aad7-af836e2224a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"
Sep 19 15:31:47.455868 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.455843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"
Sep 19 15:31:47.722310 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:47.722287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"]
Sep 19 15:31:47.724357 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:31:47.724331    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239ec181_cbcc_4f01_aad7_af836e2224a0.slice/crio-962175e45a69e9b7c73c51fbca92ebbdfe88a552d4b63bfb69b791a236c2d4d8 WatchSource:0}: Error finding container 962175e45a69e9b7c73c51fbca92ebbdfe88a552d4b63bfb69b791a236c2d4d8: Status 404 returned error can't find the container with id 962175e45a69e9b7c73c51fbca92ebbdfe88a552d4b63bfb69b791a236c2d4d8
Sep 19 15:31:48.658174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:48.658140    2579 generic.go:356] "Generic (PLEG): container finished" podID="239ec181-cbcc-4f01-aad7-af836e2224a0" containerID="284001b91877262643b4d7862e327203518f751a7dcc34b369a2bcc31bc31332" exitCode=0
Sep 19 15:31:48.658527 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:48.658212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks" event={"ID":"239ec181-cbcc-4f01-aad7-af836e2224a0","Type":"ContainerDied","Data":"284001b91877262643b4d7862e327203518f751a7dcc34b369a2bcc31bc31332"}
Sep 19 15:31:48.658527 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:48.658257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks" event={"ID":"239ec181-cbcc-4f01-aad7-af836e2224a0","Type":"ContainerStarted","Data":"962175e45a69e9b7c73c51fbca92ebbdfe88a552d4b63bfb69b791a236c2d4d8"}
Sep 19 15:31:49.789372 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:49.789351    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"
Sep 19 15:31:49.883700 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:49.883662    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/239ec181-cbcc-4f01-aad7-af836e2224a0-maas-api-service-ca\") pod \"239ec181-cbcc-4f01-aad7-af836e2224a0\" (UID: \"239ec181-cbcc-4f01-aad7-af836e2224a0\") "
Sep 19 15:31:49.883983 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:49.883961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239ec181-cbcc-4f01-aad7-af836e2224a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "239ec181-cbcc-4f01-aad7-af836e2224a0" (UID: "239ec181-cbcc-4f01-aad7-af836e2224a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:31:49.984959 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:49.984929    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/239ec181-cbcc-4f01-aad7-af836e2224a0-maas-api-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:50.669732 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:50.669699    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"
Sep 19 15:31:50.669732 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:50.669694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks" event={"ID":"239ec181-cbcc-4f01-aad7-af836e2224a0","Type":"ContainerDied","Data":"962175e45a69e9b7c73c51fbca92ebbdfe88a552d4b63bfb69b791a236c2d4d8"}
Sep 19 15:31:50.669948 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:50.669793    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="962175e45a69e9b7c73c51fbca92ebbdfe88a552d4b63bfb69b791a236c2d4d8"
Sep 19 15:31:58.770083 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:58.770049    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"]
Sep 19 15:31:58.770637 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:58.770302    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh" podUID="75eaa0a3-cbdc-407a-8c31-29e7197ec0cd" containerName="payload-processing" containerID="cri-o://2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7" gracePeriod=30
Sep 19 15:31:58.776078 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:58.776049    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"]
Sep 19 15:31:58.776305 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:58.776257    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw" podUID="4af21e20-a717-456b-8276-a204d01c57f0" containerName="payload-pre-processing" containerID="cri-o://af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649" gracePeriod=30
Sep 19 15:31:58.913751 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:58.913715    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"]
Sep 19 15:31:58.917269 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:58.917240    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a95f496f-9fxks"]
Sep 19 15:31:59.052859 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.052836    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:59.057157 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.057135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-plugins-config-volume\") pod \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\" (UID: \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\") "
Sep 19 15:31:59.057257 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.057169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsgsj\" (UniqueName: \"kubernetes.io/projected/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-kube-api-access-rsgsj\") pod \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\" (UID: \"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd\") "
Sep 19 15:31:59.057257 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.057208    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:59.057458 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.057437    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "75eaa0a3-cbdc-407a-8c31-29e7197ec0cd" (UID: "75eaa0a3-cbdc-407a-8c31-29e7197ec0cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:31:59.059338 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.059316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-kube-api-access-rsgsj" (OuterVolumeSpecName: "kube-api-access-rsgsj") pod "75eaa0a3-cbdc-407a-8c31-29e7197ec0cd" (UID: "75eaa0a3-cbdc-407a-8c31-29e7197ec0cd"). InnerVolumeSpecName "kube-api-access-rsgsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:31:59.158235 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.158206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af21e20-a717-456b-8276-a204d01c57f0-plugins-config-volume\") pod \"4af21e20-a717-456b-8276-a204d01c57f0\" (UID: \"4af21e20-a717-456b-8276-a204d01c57f0\") "
Sep 19 15:31:59.158392 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.158290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rss99\" (UniqueName: \"kubernetes.io/projected/4af21e20-a717-456b-8276-a204d01c57f0-kube-api-access-rss99\") pod \"4af21e20-a717-456b-8276-a204d01c57f0\" (UID: \"4af21e20-a717-456b-8276-a204d01c57f0\") "
Sep 19 15:31:59.158456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.158445    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:59.158496 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.158457    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsgsj\" (UniqueName: \"kubernetes.io/projected/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd-kube-api-access-rsgsj\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:59.158546 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.158524    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af21e20-a717-456b-8276-a204d01c57f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4af21e20-a717-456b-8276-a204d01c57f0" (UID: "4af21e20-a717-456b-8276-a204d01c57f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:31:59.160403 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.160382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af21e20-a717-456b-8276-a204d01c57f0-kube-api-access-rss99" (OuterVolumeSpecName: "kube-api-access-rss99") pod "4af21e20-a717-456b-8276-a204d01c57f0" (UID: "4af21e20-a717-456b-8276-a204d01c57f0"). InnerVolumeSpecName "kube-api-access-rss99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:31:59.259733 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.259704    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rss99\" (UniqueName: \"kubernetes.io/projected/4af21e20-a717-456b-8276-a204d01c57f0-kube-api-access-rss99\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:59.259733 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.259727    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4af21e20-a717-456b-8276-a204d01c57f0-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:31:59.725700 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.725661    2579 generic.go:356] "Generic (PLEG): container finished" podID="4af21e20-a717-456b-8276-a204d01c57f0" containerID="af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649" exitCode=0
Sep 19 15:31:59.725880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.725705    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"
Sep 19 15:31:59.725880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.725750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw" event={"ID":"4af21e20-a717-456b-8276-a204d01c57f0","Type":"ContainerDied","Data":"af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649"}
Sep 19 15:31:59.725880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.725810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw" event={"ID":"4af21e20-a717-456b-8276-a204d01c57f0","Type":"ContainerDied","Data":"81bb3a032822a240530619a3487b9f54c4f2ef0b515f020eb9e79764e7d647b4"}
Sep 19 15:31:59.725880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.725843    2579 scope.go:117] "RemoveContainer" containerID="af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649"
Sep 19 15:31:59.727172 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.727154    2579 generic.go:356] "Generic (PLEG): container finished" podID="75eaa0a3-cbdc-407a-8c31-29e7197ec0cd" containerID="2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7" exitCode=0
Sep 19 15:31:59.727277 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.727214    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"
Sep 19 15:31:59.727277 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.727238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh" event={"ID":"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd","Type":"ContainerDied","Data":"2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7"}
Sep 19 15:31:59.727277 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.727273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh" event={"ID":"75eaa0a3-cbdc-407a-8c31-29e7197ec0cd","Type":"ContainerDied","Data":"b5413a4237a482c60a538d74141628f0f1999bca66a0738c3b079d244c7bfe7e"}
Sep 19 15:31:59.734761 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.734740    2579 scope.go:117] "RemoveContainer" containerID="af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649"
Sep 19 15:31:59.735033 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:31:59.735006    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649\": container with ID starting with af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649 not found: ID does not exist" containerID="af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649"
Sep 19 15:31:59.735096 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.735055    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649"} err="failed to get container status \"af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649\": rpc error: code = NotFound desc = could not find container \"af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649\": container with ID starting with af0d072fdaef5c4bb038df06eef598d0784ae9fa5a552c53ae9aaa89d5d10649 not found: ID does not exist"
Sep 19 15:31:59.735096 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.735074    2579 scope.go:117] "RemoveContainer" containerID="2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7"
Sep 19 15:31:59.745391 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.745370    2579 scope.go:117] "RemoveContainer" containerID="2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7"
Sep 19 15:31:59.745689 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:31:59.745669    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7\": container with ID starting with 2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7 not found: ID does not exist" containerID="2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7"
Sep 19 15:31:59.745797 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.745694    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7"} err="failed to get container status \"2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7\": rpc error: code = NotFound desc = could not find container \"2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7\": container with ID starting with 2fb5694415de967c54b0368f33b4389792fb23dc2a3a64334df5d2f5d114f9c7 not found: ID does not exist"
Sep 19 15:31:59.758925 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.758903    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"]
Sep 19 15:31:59.763171 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.763149    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a95f496f-6b499454fd-5h8zw"]
Sep 19 15:31:59.777854 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.777824    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"]
Sep 19 15:31:59.786885 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:31:59.786853    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a95f496f-64ccbf6798-nhpgh"]
Sep 19 15:32:00.179243 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:00.179210    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="239ec181-cbcc-4f01-aad7-af836e2224a0" path="/var/lib/kubelet/pods/239ec181-cbcc-4f01-aad7-af836e2224a0/volumes"
Sep 19 15:32:00.179558 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:00.179545    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af21e20-a717-456b-8276-a204d01c57f0" path="/var/lib/kubelet/pods/4af21e20-a717-456b-8276-a204d01c57f0/volumes"
Sep 19 15:32:00.179961 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:00.179942    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75eaa0a3-cbdc-407a-8c31-29e7197ec0cd" path="/var/lib/kubelet/pods/75eaa0a3-cbdc-407a-8c31-29e7197ec0cd/volumes"
Sep 19 15:32:08.802299 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802263    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"]
Sep 19 15:32:08.802723 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802650    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af21e20-a717-456b-8276-a204d01c57f0" containerName="payload-pre-processing"
Sep 19 15:32:08.802723 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802663    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af21e20-a717-456b-8276-a204d01c57f0" containerName="payload-pre-processing"
Sep 19 15:32:08.802723 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802685    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239ec181-cbcc-4f01-aad7-af836e2224a0" containerName="revoke-keys"
Sep 19 15:32:08.802723 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802690    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="239ec181-cbcc-4f01-aad7-af836e2224a0" containerName="revoke-keys"
Sep 19 15:32:08.802723 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802697    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75eaa0a3-cbdc-407a-8c31-29e7197ec0cd" containerName="payload-processing"
Sep 19 15:32:08.802723 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802703    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="75eaa0a3-cbdc-407a-8c31-29e7197ec0cd" containerName="payload-processing"
Sep 19 15:32:08.802914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802771    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="239ec181-cbcc-4f01-aad7-af836e2224a0" containerName="revoke-keys"
Sep 19 15:32:08.802914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802780    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af21e20-a717-456b-8276-a204d01c57f0" containerName="payload-pre-processing"
Sep 19 15:32:08.802914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.802786    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="75eaa0a3-cbdc-407a-8c31-29e7197ec0cd" containerName="payload-processing"
Sep 19 15:32:08.805776 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.805761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:08.808118 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.808092    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-63d8f735-dockercfg-4czc8\""
Sep 19 15:32:08.808287 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.808270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-63d8f735\""
Sep 19 15:32:08.818515 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.818491    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"]
Sep 19 15:32:08.843136 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.843109    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"]
Sep 19 15:32:08.846424 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.846407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:08.849320 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.849296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nfm8\" (UniqueName: \"kubernetes.io/projected/e5726b0e-fac1-447d-a79a-7307b66b5440-kube-api-access-4nfm8\") pod \"payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w\" (UID: \"e5726b0e-fac1-447d-a79a-7307b66b5440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:08.849415 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.849399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5726b0e-fac1-447d-a79a-7307b66b5440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w\" (UID: \"e5726b0e-fac1-447d-a79a-7307b66b5440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:08.856923 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.856901    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"]
Sep 19 15:32:08.949943 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.949912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5726b0e-fac1-447d-a79a-7307b66b5440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w\" (UID: \"e5726b0e-fac1-447d-a79a-7307b66b5440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:08.950099 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.949951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3019b7-7a9f-4bed-99d9-c261280388a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl\" (UID: \"1e3019b7-7a9f-4bed-99d9-c261280388a5\") " pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:08.950099 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.950010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfm8\" (UniqueName: \"kubernetes.io/projected/e5726b0e-fac1-447d-a79a-7307b66b5440-kube-api-access-4nfm8\") pod \"payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w\" (UID: \"e5726b0e-fac1-447d-a79a-7307b66b5440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:08.950099 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.950026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8562f\" (UniqueName: \"kubernetes.io/projected/1e3019b7-7a9f-4bed-99d9-c261280388a5-kube-api-access-8562f\") pod \"payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl\" (UID: \"1e3019b7-7a9f-4bed-99d9-c261280388a5\") " pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:08.950575 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.950556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5726b0e-fac1-447d-a79a-7307b66b5440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w\" (UID: \"e5726b0e-fac1-447d-a79a-7307b66b5440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:08.957228 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:08.957197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nfm8\" (UniqueName: \"kubernetes.io/projected/e5726b0e-fac1-447d-a79a-7307b66b5440-kube-api-access-4nfm8\") pod \"payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w\" (UID: \"e5726b0e-fac1-447d-a79a-7307b66b5440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:09.050701 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.050658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8562f\" (UniqueName: \"kubernetes.io/projected/1e3019b7-7a9f-4bed-99d9-c261280388a5-kube-api-access-8562f\") pod \"payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl\" (UID: \"1e3019b7-7a9f-4bed-99d9-c261280388a5\") " pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:09.050863 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.050742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3019b7-7a9f-4bed-99d9-c261280388a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl\" (UID: \"1e3019b7-7a9f-4bed-99d9-c261280388a5\") " pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:09.051421 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.051403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3019b7-7a9f-4bed-99d9-c261280388a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl\" (UID: \"1e3019b7-7a9f-4bed-99d9-c261280388a5\") " pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:09.057701 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.057638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8562f\" (UniqueName: \"kubernetes.io/projected/1e3019b7-7a9f-4bed-99d9-c261280388a5-kube-api-access-8562f\") pod \"payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl\" (UID: \"1e3019b7-7a9f-4bed-99d9-c261280388a5\") " pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:09.115987 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.115961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:09.156524 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.156487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:09.406492 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.406464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"]
Sep 19 15:32:09.407762 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:32:09.407737    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5726b0e_fac1_447d_a79a_7307b66b5440.slice/crio-c8ed1fa04c979b737ed0309c0d00e824f41b27daf7659cae334f6ee0025e6123 WatchSource:0}: Error finding container c8ed1fa04c979b737ed0309c0d00e824f41b27daf7659cae334f6ee0025e6123: Status 404 returned error can't find the container with id c8ed1fa04c979b737ed0309c0d00e824f41b27daf7659cae334f6ee0025e6123
Sep 19 15:32:09.435103 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.435080    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"]
Sep 19 15:32:09.439747 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:32:09.439723    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3019b7_7a9f_4bed_99d9_c261280388a5.slice/crio-cef5c3f590141a6b0198436b96aeb41ae1d788e4bcfa6561b88dff965f2a5f5e WatchSource:0}: Error finding container cef5c3f590141a6b0198436b96aeb41ae1d788e4bcfa6561b88dff965f2a5f5e: Status 404 returned error can't find the container with id cef5c3f590141a6b0198436b96aeb41ae1d788e4bcfa6561b88dff965f2a5f5e
Sep 19 15:32:09.778503 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.778464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w" event={"ID":"e5726b0e-fac1-447d-a79a-7307b66b5440","Type":"ContainerStarted","Data":"b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640"}
Sep 19 15:32:09.778689 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.778528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w" event={"ID":"e5726b0e-fac1-447d-a79a-7307b66b5440","Type":"ContainerStarted","Data":"c8ed1fa04c979b737ed0309c0d00e824f41b27daf7659cae334f6ee0025e6123"}
Sep 19 15:32:09.778689 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.778560    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:09.780138 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.780112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl" event={"ID":"1e3019b7-7a9f-4bed-99d9-c261280388a5","Type":"ContainerStarted","Data":"e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28"}
Sep 19 15:32:09.780247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.780162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl" event={"ID":"1e3019b7-7a9f-4bed-99d9-c261280388a5","Type":"ContainerStarted","Data":"cef5c3f590141a6b0198436b96aeb41ae1d788e4bcfa6561b88dff965f2a5f5e"}
Sep 19 15:32:09.780247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.780191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:09.791937 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.791897    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w" podStartSLOduration=1.791884097 podStartE2EDuration="1.791884097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:32:09.791689682 +0000 UTC m=+1191.685253335" watchObservedRunningTime="2026-09-19 15:32:09.791884097 +0000 UTC m=+1191.685447737"
Sep 19 15:32:09.806384 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:09.806332    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl" podStartSLOduration=1.806314805 podStartE2EDuration="1.806314805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:32:09.805861192 +0000 UTC m=+1191.699424833" watchObservedRunningTime="2026-09-19 15:32:09.806314805 +0000 UTC m=+1191.699878447"
Sep 19 15:32:11.215912 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.215876    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"]
Sep 19 15:32:11.219352 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.219335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.222363 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.222330    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-15949d\"/\"default-dockercfg-c4qvm\""
Sep 19 15:32:11.222363 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.222353    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-15949d\"/\"openshift-service-ca.crt\""
Sep 19 15:32:11.222516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.222332    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-15949d\"/\"premium-w0-15949d-kserve-self-signed-certs\""
Sep 19 15:32:11.222516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.222330    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-15949d\"/\"kube-root-ca.crt\""
Sep 19 15:32:11.227417 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.227386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"]
Sep 19 15:32:11.270702 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.270672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-model-cache\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.270841 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.270737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-dshm\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.270841 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.270760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e132ed7-f33a-424b-abbf-4d7755336f99-tls-certs\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.270841 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.270786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qdf\" (UniqueName: \"kubernetes.io/projected/6e132ed7-f33a-424b-abbf-4d7755336f99-kube-api-access-w7qdf\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.270998 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.270887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-home\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.270998 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.270922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-tmp-dir\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371350 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qdf\" (UniqueName: \"kubernetes.io/projected/6e132ed7-f33a-424b-abbf-4d7755336f99-kube-api-access-w7qdf\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371520 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-home\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371520 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-tmp-dir\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371520 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-model-cache\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371673 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-dshm\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371673 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e132ed7-f33a-424b-abbf-4d7755336f99-tls-certs\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371841 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-model-cache\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-home\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.371931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.371913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-tmp-dir\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.373870 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.373850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-dshm\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.374140 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.374124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e132ed7-f33a-424b-abbf-4d7755336f99-tls-certs\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.379400 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.379383    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qdf\" (UniqueName: \"kubernetes.io/projected/6e132ed7-f33a-424b-abbf-4d7755336f99-kube-api-access-w7qdf\") pod \"premium-w0-15949d-kserve-765967dff4-6l986\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") " pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.529693 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.529589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:11.799928 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:11.799897    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"]
Sep 19 15:32:11.801459 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:32:11.801432    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e132ed7_f33a_424b_abbf_4d7755336f99.slice/crio-ec332e9053ade703670223e838edd253dfceb6c2b25bcda29fa82df4ee42b184 WatchSource:0}: Error finding container ec332e9053ade703670223e838edd253dfceb6c2b25bcda29fa82df4ee42b184: Status 404 returned error can't find the container with id ec332e9053ade703670223e838edd253dfceb6c2b25bcda29fa82df4ee42b184
Sep 19 15:32:12.798029 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:12.797986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986" event={"ID":"6e132ed7-f33a-424b-abbf-4d7755336f99","Type":"ContainerStarted","Data":"cc22bf6920e65709548afc8795952e4cfb86708662ca4eaa36231181a1cccbdd"}
Sep 19 15:32:12.798410 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:12.798038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986" event={"ID":"6e132ed7-f33a-424b-abbf-4d7755336f99","Type":"ContainerStarted","Data":"ec332e9053ade703670223e838edd253dfceb6c2b25bcda29fa82df4ee42b184"}
Sep 19 15:32:12.813032 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:12.812981    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986" podStartSLOduration=1.8129645989999998 podStartE2EDuration="1.812964599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:32:12.811153873 +0000 UTC m=+1194.704717515" watchObservedRunningTime="2026-09-19 15:32:12.812964599 +0000 UTC m=+1194.706528239"
Sep 19 15:32:20.162308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:20.162280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:32:20.163097 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:20.163078    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:32:20.788297 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:20.788262    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:20.788472 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:20.788455    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:21.530113 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:21.530077    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:21.530587 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:21.530130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:21.541932 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:21.541906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:21.860175 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:21.860093    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:32:36.410007 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.409969    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"]
Sep 19 15:32:36.413433 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.413410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"
Sep 19 15:32:36.415490 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.415469    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7znsl\""
Sep 19 15:32:36.415587 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.415490    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 15:32:36.419226 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.419198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"]
Sep 19 15:32:36.607412 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.607364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ebc643-94b5-421c-9957-98cef1c82540-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n\" (UID: \"f7ebc643-94b5-421c-9957-98cef1c82540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"
Sep 19 15:32:36.708785 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.708741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ebc643-94b5-421c-9957-98cef1c82540-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n\" (UID: \"f7ebc643-94b5-421c-9957-98cef1c82540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"
Sep 19 15:32:36.709373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.709351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ebc643-94b5-421c-9957-98cef1c82540-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n\" (UID: \"f7ebc643-94b5-421c-9957-98cef1c82540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"
Sep 19 15:32:36.725684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.725667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"
Sep 19 15:32:36.991695 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:36.991672    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"]
Sep 19 15:32:36.993337 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:32:36.993309    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ebc643_94b5_421c_9957_98cef1c82540.slice/crio-c39c44733ba01aa050762a107494eb7a500efe9a96a5b7c4250af2093f7757d6 WatchSource:0}: Error finding container c39c44733ba01aa050762a107494eb7a500efe9a96a5b7c4250af2093f7757d6: Status 404 returned error can't find the container with id c39c44733ba01aa050762a107494eb7a500efe9a96a5b7c4250af2093f7757d6
Sep 19 15:32:37.930247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:37.930215    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7ebc643-94b5-421c-9957-98cef1c82540" containerID="be772ea8356ebd53fc9447687aabe8db535407ba5b0af60e3cb9ec26152d93b2" exitCode=0
Sep 19 15:32:37.930629 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:37.930302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n" event={"ID":"f7ebc643-94b5-421c-9957-98cef1c82540","Type":"ContainerDied","Data":"be772ea8356ebd53fc9447687aabe8db535407ba5b0af60e3cb9ec26152d93b2"}
Sep 19 15:32:37.930629 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:37.930349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n" event={"ID":"f7ebc643-94b5-421c-9957-98cef1c82540","Type":"ContainerStarted","Data":"c39c44733ba01aa050762a107494eb7a500efe9a96a5b7c4250af2093f7757d6"}
Sep 19 15:32:39.061282 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:39.061259    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"
Sep 19 15:32:39.136729 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:39.136694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ebc643-94b5-421c-9957-98cef1c82540-maas-api-service-ca\") pod \"f7ebc643-94b5-421c-9957-98cef1c82540\" (UID: \"f7ebc643-94b5-421c-9957-98cef1c82540\") "
Sep 19 15:32:39.137031 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:39.137009    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ebc643-94b5-421c-9957-98cef1c82540-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7ebc643-94b5-421c-9957-98cef1c82540" (UID: "f7ebc643-94b5-421c-9957-98cef1c82540"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:32:39.237696 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:39.237594    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7ebc643-94b5-421c-9957-98cef1c82540-maas-api-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:32:39.941371 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:39.941334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n" event={"ID":"f7ebc643-94b5-421c-9957-98cef1c82540","Type":"ContainerDied","Data":"c39c44733ba01aa050762a107494eb7a500efe9a96a5b7c4250af2093f7757d6"}
Sep 19 15:32:39.941371 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:39.941368    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"
Sep 19 15:32:39.941586 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:39.941382    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c39c44733ba01aa050762a107494eb7a500efe9a96a5b7c4250af2093f7757d6"
Sep 19 15:32:47.255052 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.255020    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"]
Sep 19 15:32:47.255531 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.255225    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl" podUID="1e3019b7-7a9f-4bed-99d9-c261280388a5" containerName="payload-processing" containerID="cri-o://e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28" gracePeriod=30
Sep 19 15:32:47.260408 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.260383    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"]
Sep 19 15:32:47.260648 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.260621    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w" podUID="e5726b0e-fac1-447d-a79a-7307b66b5440" containerName="payload-pre-processing" containerID="cri-o://b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640" gracePeriod=30
Sep 19 15:32:47.405164 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.405122    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"]
Sep 19 15:32:47.407569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.407535    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d8f735-s8j6n"]
Sep 19 15:32:47.513370 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.513309    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:47.543078 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.543057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:47.606761 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.606733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8562f\" (UniqueName: \"kubernetes.io/projected/1e3019b7-7a9f-4bed-99d9-c261280388a5-kube-api-access-8562f\") pod \"1e3019b7-7a9f-4bed-99d9-c261280388a5\" (UID: \"1e3019b7-7a9f-4bed-99d9-c261280388a5\") "
Sep 19 15:32:47.606929 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.606793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3019b7-7a9f-4bed-99d9-c261280388a5-plugins-config-volume\") pod \"1e3019b7-7a9f-4bed-99d9-c261280388a5\" (UID: \"1e3019b7-7a9f-4bed-99d9-c261280388a5\") "
Sep 19 15:32:47.606929 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.606825    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5726b0e-fac1-447d-a79a-7307b66b5440-plugins-config-volume\") pod \"e5726b0e-fac1-447d-a79a-7307b66b5440\" (UID: \"e5726b0e-fac1-447d-a79a-7307b66b5440\") "
Sep 19 15:32:47.607023 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.606938    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nfm8\" (UniqueName: \"kubernetes.io/projected/e5726b0e-fac1-447d-a79a-7307b66b5440-kube-api-access-4nfm8\") pod \"e5726b0e-fac1-447d-a79a-7307b66b5440\" (UID: \"e5726b0e-fac1-447d-a79a-7307b66b5440\") "
Sep 19 15:32:47.607160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.607140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5726b0e-fac1-447d-a79a-7307b66b5440-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5726b0e-fac1-447d-a79a-7307b66b5440" (UID: "e5726b0e-fac1-447d-a79a-7307b66b5440"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:32:47.607248 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.607180    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3019b7-7a9f-4bed-99d9-c261280388a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e3019b7-7a9f-4bed-99d9-c261280388a5" (UID: "1e3019b7-7a9f-4bed-99d9-c261280388a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:32:47.607348 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.607325    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3019b7-7a9f-4bed-99d9-c261280388a5-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:32:47.607348 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.607344    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5726b0e-fac1-447d-a79a-7307b66b5440-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:32:47.609132 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.609109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3019b7-7a9f-4bed-99d9-c261280388a5-kube-api-access-8562f" (OuterVolumeSpecName: "kube-api-access-8562f") pod "1e3019b7-7a9f-4bed-99d9-c261280388a5" (UID: "1e3019b7-7a9f-4bed-99d9-c261280388a5"). InnerVolumeSpecName "kube-api-access-8562f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:32:47.609233 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.609128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5726b0e-fac1-447d-a79a-7307b66b5440-kube-api-access-4nfm8" (OuterVolumeSpecName: "kube-api-access-4nfm8") pod "e5726b0e-fac1-447d-a79a-7307b66b5440" (UID: "e5726b0e-fac1-447d-a79a-7307b66b5440"). InnerVolumeSpecName "kube-api-access-4nfm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:32:47.707779 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.707748    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8562f\" (UniqueName: \"kubernetes.io/projected/1e3019b7-7a9f-4bed-99d9-c261280388a5-kube-api-access-8562f\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:32:47.707779 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.707772    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nfm8\" (UniqueName: \"kubernetes.io/projected/e5726b0e-fac1-447d-a79a-7307b66b5440-kube-api-access-4nfm8\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:32:47.981574 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.981538    2579 generic.go:356] "Generic (PLEG): container finished" podID="1e3019b7-7a9f-4bed-99d9-c261280388a5" containerID="e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28" exitCode=0
Sep 19 15:32:47.981953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.981584    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"
Sep 19 15:32:47.981953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.981624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl" event={"ID":"1e3019b7-7a9f-4bed-99d9-c261280388a5","Type":"ContainerDied","Data":"e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28"}
Sep 19 15:32:47.981953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.981676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl" event={"ID":"1e3019b7-7a9f-4bed-99d9-c261280388a5","Type":"ContainerDied","Data":"cef5c3f590141a6b0198436b96aeb41ae1d788e4bcfa6561b88dff965f2a5f5e"}
Sep 19 15:32:47.981953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.981717    2579 scope.go:117] "RemoveContainer" containerID="e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28"
Sep 19 15:32:47.982991 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.982968    2579 generic.go:356] "Generic (PLEG): container finished" podID="e5726b0e-fac1-447d-a79a-7307b66b5440" containerID="b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640" exitCode=0
Sep 19 15:32:47.983083 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.983005    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"
Sep 19 15:32:47.983083 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.983037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w" event={"ID":"e5726b0e-fac1-447d-a79a-7307b66b5440","Type":"ContainerDied","Data":"b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640"}
Sep 19 15:32:47.983083 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.983072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w" event={"ID":"e5726b0e-fac1-447d-a79a-7307b66b5440","Type":"ContainerDied","Data":"c8ed1fa04c979b737ed0309c0d00e824f41b27daf7659cae334f6ee0025e6123"}
Sep 19 15:32:47.991561 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.991542    2579 scope.go:117] "RemoveContainer" containerID="e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28"
Sep 19 15:32:47.991844 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:32:47.991823    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28\": container with ID starting with e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28 not found: ID does not exist" containerID="e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28"
Sep 19 15:32:47.991899 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.991858    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28"} err="failed to get container status \"e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28\": rpc error: code = NotFound desc = could not find container \"e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28\": container with ID starting with e111aa7ef782c7c3feb7c1c098dfc953e25cf61edfa3be38261dd242feb17f28 not found: ID does not exist"
Sep 19 15:32:47.991899 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.991881    2579 scope.go:117] "RemoveContainer" containerID="b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640"
Sep 19 15:32:47.999568 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.999552    2579 scope.go:117] "RemoveContainer" containerID="b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640"
Sep 19 15:32:47.999855 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:32:47.999833    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640\": container with ID starting with b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640 not found: ID does not exist" containerID="b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640"
Sep 19 15:32:47.999933 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:47.999863    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640"} err="failed to get container status \"b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640\": rpc error: code = NotFound desc = could not find container \"b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640\": container with ID starting with b874115c89b07f9c85a81709b5dbce1ed66e1f054680def7cc799f9e4d19d640 not found: ID does not exist"
Sep 19 15:32:48.004619 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:48.004581    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"]
Sep 19 15:32:48.008118 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:48.008097    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63d8f735-c78df56c8-l9h9w"]
Sep 19 15:32:48.019396 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:48.019376    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"]
Sep 19 15:32:48.027968 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:48.024970    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63d8f735-7775688b7f-wg5kl"]
Sep 19 15:32:48.171232 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:48.171188    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3019b7-7a9f-4bed-99d9-c261280388a5" path="/var/lib/kubelet/pods/1e3019b7-7a9f-4bed-99d9-c261280388a5/volumes"
Sep 19 15:32:48.171570 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:48.171556    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5726b0e-fac1-447d-a79a-7307b66b5440" path="/var/lib/kubelet/pods/e5726b0e-fac1-447d-a79a-7307b66b5440/volumes"
Sep 19 15:32:48.171855 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:48.171843    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7ebc643-94b5-421c-9957-98cef1c82540" path="/var/lib/kubelet/pods/f7ebc643-94b5-421c-9957-98cef1c82540/volumes"
Sep 19 15:32:55.486078 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486045    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"]
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486403    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5726b0e-fac1-447d-a79a-7307b66b5440" containerName="payload-pre-processing"
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486413    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5726b0e-fac1-447d-a79a-7307b66b5440" containerName="payload-pre-processing"
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486421    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7ebc643-94b5-421c-9957-98cef1c82540" containerName="revoke-keys"
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486426    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7ebc643-94b5-421c-9957-98cef1c82540" containerName="revoke-keys"
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486434    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3019b7-7a9f-4bed-99d9-c261280388a5" containerName="payload-processing"
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486439    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3019b7-7a9f-4bed-99d9-c261280388a5" containerName="payload-processing"
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486506    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3019b7-7a9f-4bed-99d9-c261280388a5" containerName="payload-processing"
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486514    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7ebc643-94b5-421c-9957-98cef1c82540" containerName="revoke-keys"
Sep 19 15:32:55.486552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.486522    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5726b0e-fac1-447d-a79a-7307b66b5440" containerName="payload-pre-processing"
Sep 19 15:32:55.491032 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.491014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:55.493208 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.493185    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-24d51017\""
Sep 19 15:32:55.493342 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.493234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-24d51017-dockercfg-b95cf\""
Sep 19 15:32:55.498430 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.498400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"]
Sep 19 15:32:55.519378 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.519349    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"]
Sep 19 15:32:55.522738 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.522722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:55.530858 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.530832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"]
Sep 19 15:32:55.577978 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.577943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmth\" (UniqueName: \"kubernetes.io/projected/ce686fa5-457e-4df8-9d37-22f996c08440-kube-api-access-dqmth\") pod \"payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q\" (UID: \"ce686fa5-457e-4df8-9d37-22f996c08440\") " pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:55.578132 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.578011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce686fa5-457e-4df8-9d37-22f996c08440-plugins-config-volume\") pod \"payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q\" (UID: \"ce686fa5-457e-4df8-9d37-22f996c08440\") " pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:55.578132 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.578082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cnx5\" (UniqueName: \"kubernetes.io/projected/f01d174b-2856-4eee-bff3-9b9a9ba7209f-kube-api-access-5cnx5\") pod \"payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x\" (UID: \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:55.578132 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.578110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01d174b-2856-4eee-bff3-9b9a9ba7209f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x\" (UID: \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:55.679514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.679474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cnx5\" (UniqueName: \"kubernetes.io/projected/f01d174b-2856-4eee-bff3-9b9a9ba7209f-kube-api-access-5cnx5\") pod \"payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x\" (UID: \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:55.679514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.679515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01d174b-2856-4eee-bff3-9b9a9ba7209f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x\" (UID: \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:55.679775 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.679562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmth\" (UniqueName: \"kubernetes.io/projected/ce686fa5-457e-4df8-9d37-22f996c08440-kube-api-access-dqmth\") pod \"payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q\" (UID: \"ce686fa5-457e-4df8-9d37-22f996c08440\") " pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:55.679775 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.679636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce686fa5-457e-4df8-9d37-22f996c08440-plugins-config-volume\") pod \"payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q\" (UID: \"ce686fa5-457e-4df8-9d37-22f996c08440\") " pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:55.680318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.680297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01d174b-2856-4eee-bff3-9b9a9ba7209f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x\" (UID: \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:55.680363 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.680305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce686fa5-457e-4df8-9d37-22f996c08440-plugins-config-volume\") pod \"payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q\" (UID: \"ce686fa5-457e-4df8-9d37-22f996c08440\") " pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:55.687169 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.687145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmth\" (UniqueName: \"kubernetes.io/projected/ce686fa5-457e-4df8-9d37-22f996c08440-kube-api-access-dqmth\") pod \"payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q\" (UID: \"ce686fa5-457e-4df8-9d37-22f996c08440\") " pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:55.687313 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.687293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cnx5\" (UniqueName: \"kubernetes.io/projected/f01d174b-2856-4eee-bff3-9b9a9ba7209f-kube-api-access-5cnx5\") pod \"payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x\" (UID: \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:55.801404 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.801322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:55.833697 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:55.833660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:56.074996 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.074970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"]
Sep 19 15:32:56.077127 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:32:56.077104    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01d174b_2856_4eee_bff3_9b9a9ba7209f.slice/crio-a1666a0651535f980ac083f22c5e6222d36dca262c5d4bbfbf57c558c15dcc2c WatchSource:0}: Error finding container a1666a0651535f980ac083f22c5e6222d36dca262c5d4bbfbf57c558c15dcc2c: Status 404 returned error can't find the container with id a1666a0651535f980ac083f22c5e6222d36dca262c5d4bbfbf57c558c15dcc2c
Sep 19 15:32:56.108923 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.108149    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"]
Sep 19 15:32:56.111010 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:32:56.110987    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce686fa5_457e_4df8_9d37_22f996c08440.slice/crio-68ba03e3822f7cbe9486840db34078fd37e920b7afb65d9fdfd5dc6dc677b029 WatchSource:0}: Error finding container 68ba03e3822f7cbe9486840db34078fd37e920b7afb65d9fdfd5dc6dc677b029: Status 404 returned error can't find the container with id 68ba03e3822f7cbe9486840db34078fd37e920b7afb65d9fdfd5dc6dc677b029
Sep 19 15:32:56.513840 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.513809    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"]
Sep 19 15:32:56.517247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.517227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.520086 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.520042    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e93f57\"/\"default-dockercfg-kcvck\""
Sep 19 15:32:56.520086 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.520060    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e93f57\"/\"kube-root-ca.crt\""
Sep 19 15:32:56.520277 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.520175    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e93f57\"/\"test-model-e93f57-kserve-self-signed-certs\""
Sep 19 15:32:56.520277 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.520195    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e93f57\"/\"openshift-service-ca.crt\""
Sep 19 15:32:56.524469 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.524446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"]
Sep 19 15:32:56.587971 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.587942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e621478-2084-4944-994d-e84aa84c9a9e-tls-certs\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.588140 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.588005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-home\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.588140 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.588071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-dshm\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.588140 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.588132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-model-cache\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.588332 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.588169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkphd\" (UniqueName: \"kubernetes.io/projected/3e621478-2084-4944-994d-e84aa84c9a9e-kube-api-access-lkphd\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.588332 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.588190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-tmp-dir\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.689769 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.689739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-home\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.689959 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.689780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-dshm\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.689959 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.689817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-model-cache\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.689959 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.689853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkphd\" (UniqueName: \"kubernetes.io/projected/3e621478-2084-4944-994d-e84aa84c9a9e-kube-api-access-lkphd\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.689959 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.689876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-tmp-dir\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.689959 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.689895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e621478-2084-4944-994d-e84aa84c9a9e-tls-certs\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.690311 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.690285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-home\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.690371 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.690352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-model-cache\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.690574 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.690526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-tmp-dir\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.692159 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.692140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-dshm\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.692959 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.692941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e621478-2084-4944-994d-e84aa84c9a9e-tls-certs\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.702091 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.702068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkphd\" (UniqueName: \"kubernetes.io/projected/3e621478-2084-4944-994d-e84aa84c9a9e-kube-api-access-lkphd\") pod \"test-model-e93f57-kserve-7b9fffdd59-k828d\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") " pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:56.827876 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:56.827785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:32:57.028137 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.028100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q" event={"ID":"ce686fa5-457e-4df8-9d37-22f996c08440","Type":"ContainerStarted","Data":"cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a"}
Sep 19 15:32:57.028327 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.028167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q" event={"ID":"ce686fa5-457e-4df8-9d37-22f996c08440","Type":"ContainerStarted","Data":"68ba03e3822f7cbe9486840db34078fd37e920b7afb65d9fdfd5dc6dc677b029"}
Sep 19 15:32:57.028327 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.028295    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:32:57.029769 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.029749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x" event={"ID":"f01d174b-2856-4eee-bff3-9b9a9ba7209f","Type":"ContainerStarted","Data":"3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5"}
Sep 19 15:32:57.029861 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.029782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x" event={"ID":"f01d174b-2856-4eee-bff3-9b9a9ba7209f","Type":"ContainerStarted","Data":"a1666a0651535f980ac083f22c5e6222d36dca262c5d4bbfbf57c558c15dcc2c"}
Sep 19 15:32:57.029899 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.029880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:32:57.042878 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.042830    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q" podStartSLOduration=2.042808021 podStartE2EDuration="2.042808021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:32:57.042409915 +0000 UTC m=+1238.935973557" watchObservedRunningTime="2026-09-19 15:32:57.042808021 +0000 UTC m=+1238.936371662"
Sep 19 15:32:57.057884 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.057838    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x" podStartSLOduration=2.057825942 podStartE2EDuration="2.057825942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:32:57.055462462 +0000 UTC m=+1238.949026102" watchObservedRunningTime="2026-09-19 15:32:57.057825942 +0000 UTC m=+1238.951389592"
Sep 19 15:32:57.101091 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:57.101068    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"]
Sep 19 15:32:57.103297 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:32:57.103268    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e621478_2084_4944_994d_e84aa84c9a9e.slice/crio-ddd44a0470bde7a78ec584aa612faffe9748aba016c18c569ffe21ac061b199a WatchSource:0}: Error finding container ddd44a0470bde7a78ec584aa612faffe9748aba016c18c569ffe21ac061b199a: Status 404 returned error can't find the container with id ddd44a0470bde7a78ec584aa612faffe9748aba016c18c569ffe21ac061b199a
Sep 19 15:32:58.036195 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:58.036155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d" event={"ID":"3e621478-2084-4944-994d-e84aa84c9a9e","Type":"ContainerStarted","Data":"286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426"}
Sep 19 15:32:58.036576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:58.036203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d" event={"ID":"3e621478-2084-4944-994d-e84aa84c9a9e","Type":"ContainerStarted","Data":"ddd44a0470bde7a78ec584aa612faffe9748aba016c18c569ffe21ac061b199a"}
Sep 19 15:32:58.050384 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:32:58.050334    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d" podStartSLOduration=2.050320117 podStartE2EDuration="2.050320117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:32:58.049734557 +0000 UTC m=+1239.943298220" watchObservedRunningTime="2026-09-19 15:32:58.050320117 +0000 UTC m=+1239.943883757"
Sep 19 15:33:06.828518 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:06.828483    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:33:06.828911 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:06.828760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:33:06.841106 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:06.841074    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:33:07.093665 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:07.093570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:33:08.038176 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:08.038147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:33:08.038573 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:08.038217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:33:33.652484 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:33.652450    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"]
Sep 19 15:33:33.653054 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:33.652718    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q" podUID="ce686fa5-457e-4df8-9d37-22f996c08440" containerName="payload-processing" containerID="cri-o://cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a" gracePeriod=30
Sep 19 15:33:33.656978 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:33.656939    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"]
Sep 19 15:33:33.657248 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:33.657190    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x" podUID="f01d174b-2856-4eee-bff3-9b9a9ba7209f" containerName="payload-pre-processing" containerID="cri-o://3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5" gracePeriod=30
Sep 19 15:33:33.944066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:33.944044    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:33:33.946883 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:33.946862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:33:34.034160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.034131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01d174b-2856-4eee-bff3-9b9a9ba7209f-plugins-config-volume\") pod \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\" (UID: \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\") "
Sep 19 15:33:34.034309 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.034170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce686fa5-457e-4df8-9d37-22f996c08440-plugins-config-volume\") pod \"ce686fa5-457e-4df8-9d37-22f996c08440\" (UID: \"ce686fa5-457e-4df8-9d37-22f996c08440\") "
Sep 19 15:33:34.034309 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.034264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqmth\" (UniqueName: \"kubernetes.io/projected/ce686fa5-457e-4df8-9d37-22f996c08440-kube-api-access-dqmth\") pod \"ce686fa5-457e-4df8-9d37-22f996c08440\" (UID: \"ce686fa5-457e-4df8-9d37-22f996c08440\") "
Sep 19 15:33:34.034309 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.034280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cnx5\" (UniqueName: \"kubernetes.io/projected/f01d174b-2856-4eee-bff3-9b9a9ba7209f-kube-api-access-5cnx5\") pod \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\" (UID: \"f01d174b-2856-4eee-bff3-9b9a9ba7209f\") "
Sep 19 15:33:34.034534 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.034512    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01d174b-2856-4eee-bff3-9b9a9ba7209f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f01d174b-2856-4eee-bff3-9b9a9ba7209f" (UID: "f01d174b-2856-4eee-bff3-9b9a9ba7209f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:33:34.034588 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.034567    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce686fa5-457e-4df8-9d37-22f996c08440-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce686fa5-457e-4df8-9d37-22f996c08440" (UID: "ce686fa5-457e-4df8-9d37-22f996c08440"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:33:34.036532 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.036506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce686fa5-457e-4df8-9d37-22f996c08440-kube-api-access-dqmth" (OuterVolumeSpecName: "kube-api-access-dqmth") pod "ce686fa5-457e-4df8-9d37-22f996c08440" (UID: "ce686fa5-457e-4df8-9d37-22f996c08440"). InnerVolumeSpecName "kube-api-access-dqmth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:33:34.036651 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.036522    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01d174b-2856-4eee-bff3-9b9a9ba7209f-kube-api-access-5cnx5" (OuterVolumeSpecName: "kube-api-access-5cnx5") pod "f01d174b-2856-4eee-bff3-9b9a9ba7209f" (UID: "f01d174b-2856-4eee-bff3-9b9a9ba7209f"). InnerVolumeSpecName "kube-api-access-5cnx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:33:34.135596 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.135571    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f01d174b-2856-4eee-bff3-9b9a9ba7209f-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:33:34.135731 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.135612    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce686fa5-457e-4df8-9d37-22f996c08440-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:33:34.135731 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.135625    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqmth\" (UniqueName: \"kubernetes.io/projected/ce686fa5-457e-4df8-9d37-22f996c08440-kube-api-access-dqmth\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:33:34.135731 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.135633    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cnx5\" (UniqueName: \"kubernetes.io/projected/f01d174b-2856-4eee-bff3-9b9a9ba7209f-kube-api-access-5cnx5\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:33:34.226746 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.226720    2579 generic.go:356] "Generic (PLEG): container finished" podID="ce686fa5-457e-4df8-9d37-22f996c08440" containerID="cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a" exitCode=0
Sep 19 15:33:34.226880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.226763    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"
Sep 19 15:33:34.226880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.226813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q" event={"ID":"ce686fa5-457e-4df8-9d37-22f996c08440","Type":"ContainerDied","Data":"cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a"}
Sep 19 15:33:34.226880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.226864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q" event={"ID":"ce686fa5-457e-4df8-9d37-22f996c08440","Type":"ContainerDied","Data":"68ba03e3822f7cbe9486840db34078fd37e920b7afb65d9fdfd5dc6dc677b029"}
Sep 19 15:33:34.226995 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.226890    2579 scope.go:117] "RemoveContainer" containerID="cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a"
Sep 19 15:33:34.228528 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.228505    2579 generic.go:356] "Generic (PLEG): container finished" podID="f01d174b-2856-4eee-bff3-9b9a9ba7209f" containerID="3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5" exitCode=0
Sep 19 15:33:34.228658 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.228540    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"
Sep 19 15:33:34.228658 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.228566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x" event={"ID":"f01d174b-2856-4eee-bff3-9b9a9ba7209f","Type":"ContainerDied","Data":"3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5"}
Sep 19 15:33:34.228658 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.228643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x" event={"ID":"f01d174b-2856-4eee-bff3-9b9a9ba7209f","Type":"ContainerDied","Data":"a1666a0651535f980ac083f22c5e6222d36dca262c5d4bbfbf57c558c15dcc2c"}
Sep 19 15:33:34.236262 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.236248    2579 scope.go:117] "RemoveContainer" containerID="cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a"
Sep 19 15:33:34.236523 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:33:34.236505    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a\": container with ID starting with cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a not found: ID does not exist" containerID="cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a"
Sep 19 15:33:34.236595 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.236532    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a"} err="failed to get container status \"cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a\": rpc error: code = NotFound desc = could not find container \"cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a\": container with ID starting with cf0c6ebc6a7fc8e68ca2bf9a2ae83ac3e09c09406145ac5df8acc8090b9e9e4a not found: ID does not exist"
Sep 19 15:33:34.236595 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.236554    2579 scope.go:117] "RemoveContainer" containerID="3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5"
Sep 19 15:33:34.244543 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.244526    2579 scope.go:117] "RemoveContainer" containerID="3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5"
Sep 19 15:33:34.244830 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:33:34.244812    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5\": container with ID starting with 3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5 not found: ID does not exist" containerID="3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5"
Sep 19 15:33:34.244904 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.244836    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5"} err="failed to get container status \"3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5\": rpc error: code = NotFound desc = could not find container \"3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5\": container with ID starting with 3887e4e60fa3ed134cef6812426bf5aa54c7fddb2df4bb666af1408fcd16adf5 not found: ID does not exist"
Sep 19 15:33:34.246746 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.246724    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"]
Sep 19 15:33:34.252057 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.252039    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-24d51017-6b5b6d67df-ht92q"]
Sep 19 15:33:34.267115 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.267092    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"]
Sep 19 15:33:34.273986 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:34.273966    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-24d51017-8596956b98-6426x"]
Sep 19 15:33:36.170645 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:36.170592    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce686fa5-457e-4df8-9d37-22f996c08440" path="/var/lib/kubelet/pods/ce686fa5-457e-4df8-9d37-22f996c08440/volumes"
Sep 19 15:33:36.171014 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:36.170965    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f01d174b-2856-4eee-bff3-9b9a9ba7209f" path="/var/lib/kubelet/pods/f01d174b-2856-4eee-bff3-9b9a9ba7209f/volumes"
Sep 19 15:33:49.926339 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.926306    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"]
Sep 19 15:33:49.926788 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.926711    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f01d174b-2856-4eee-bff3-9b9a9ba7209f" containerName="payload-pre-processing"
Sep 19 15:33:49.926788 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.926737    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f01d174b-2856-4eee-bff3-9b9a9ba7209f" containerName="payload-pre-processing"
Sep 19 15:33:49.926788 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.926757    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce686fa5-457e-4df8-9d37-22f996c08440" containerName="payload-processing"
Sep 19 15:33:49.926788 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.926766    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce686fa5-457e-4df8-9d37-22f996c08440" containerName="payload-processing"
Sep 19 15:33:49.926968 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.926861    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce686fa5-457e-4df8-9d37-22f996c08440" containerName="payload-processing"
Sep 19 15:33:49.926968 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.926878    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f01d174b-2856-4eee-bff3-9b9a9ba7209f" containerName="payload-pre-processing"
Sep 19 15:33:49.929877 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.929858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:49.931956 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.931935    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-15949d\"/\"distinct-w0-15949d-kserve-self-signed-certs\""
Sep 19 15:33:49.939300 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.939275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"]
Sep 19 15:33:49.981427 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.981402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-dshm\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:49.981558 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.981472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-tmp-dir\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:49.981558 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.981531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-model-cache\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:49.981685 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.981563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbnm\" (UniqueName: \"kubernetes.io/projected/c7ea8b18-5c80-43db-8394-a9084e396311-kube-api-access-xvbnm\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:49.981685 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.981636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-home\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:49.981685 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:49.981662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ea8b18-5c80-43db-8394-a9084e396311-tls-certs\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.082660 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.082628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-dshm\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.082809 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.082694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-tmp-dir\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.082809 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.082731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-model-cache\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.082809 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.082752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbnm\" (UniqueName: \"kubernetes.io/projected/c7ea8b18-5c80-43db-8394-a9084e396311-kube-api-access-xvbnm\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.082809 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.082778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-home\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.082809 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.082797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ea8b18-5c80-43db-8394-a9084e396311-tls-certs\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.083125 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.083104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-tmp-dir\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.083167 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.083117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-model-cache\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.083200 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.083175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-home\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.085123 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.085095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-dshm\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.085341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.085324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ea8b18-5c80-43db-8394-a9084e396311-tls-certs\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.089751 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.089733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbnm\" (UniqueName: \"kubernetes.io/projected/c7ea8b18-5c80-43db-8394-a9084e396311-kube-api-access-xvbnm\") pod \"distinct-w0-15949d-kserve-57778947fc-427ds\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") " pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.241318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.241227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:33:50.508038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:50.508014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"]
Sep 19 15:33:50.510537 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:33:50.510499    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7ea8b18_5c80_43db_8394_a9084e396311.slice/crio-7f3b5a2d11c0b6cfe394533d4c46e56cb3ec95808faa8818ddd11724671a6bba WatchSource:0}: Error finding container 7f3b5a2d11c0b6cfe394533d4c46e56cb3ec95808faa8818ddd11724671a6bba: Status 404 returned error can't find the container with id 7f3b5a2d11c0b6cfe394533d4c46e56cb3ec95808faa8818ddd11724671a6bba
Sep 19 15:33:51.314172 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:51.314137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds" event={"ID":"c7ea8b18-5c80-43db-8394-a9084e396311","Type":"ContainerStarted","Data":"c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367"}
Sep 19 15:33:51.314172 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:51.314180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds" event={"ID":"c7ea8b18-5c80-43db-8394-a9084e396311","Type":"ContainerStarted","Data":"7f3b5a2d11c0b6cfe394533d4c46e56cb3ec95808faa8818ddd11724671a6bba"}
Sep 19 15:33:51.330510 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:33:51.330461    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds" podStartSLOduration=2.330446062 podStartE2EDuration="2.330446062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:33:51.328149322 +0000 UTC m=+1293.221712965" watchObservedRunningTime="2026-09-19 15:33:51.330446062 +0000 UTC m=+1293.224009746"
Sep 19 15:34:00.242175 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:00.242135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:34:00.242641 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:00.242195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:34:00.254545 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:00.254518    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:34:00.370373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:00.370348    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:34:18.935399 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:18.935369    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"]
Sep 19 15:34:18.938844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:18.938825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:18.940901 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:18.940882    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-33737b\"/\"distinct-2-w3-33737b-kserve-self-signed-certs\""
Sep 19 15:34:18.946318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:18.946176    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"]
Sep 19 15:34:19.044199 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.044172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tls-certs\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.044326 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.044204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hqhl\" (UniqueName: \"kubernetes.io/projected/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-kube-api-access-9hqhl\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.044326 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.044237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-model-cache\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.044451 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.044339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tmp-dir\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.044451 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.044411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-home\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.044451 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.044438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-dshm\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145260 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tls-certs\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqhl\" (UniqueName: \"kubernetes.io/projected/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-kube-api-access-9hqhl\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145373 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-model-cache\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145470 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tmp-dir\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145470 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-home\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145470 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-dshm\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145816 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tmp-dir\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145888 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-model-cache\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.145925 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.145881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-home\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.147869 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.147846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-dshm\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.148019 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.148003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tls-certs\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.154988 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.154957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hqhl\" (UniqueName: \"kubernetes.io/projected/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-kube-api-access-9hqhl\") pod \"distinct-2-w3-33737b-kserve-8694b4565d-md7tm\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") " pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.248378 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.248333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:19.514783 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:19.514754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"]
Sep 19 15:34:19.516968 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:34:19.516942    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32dd25e7_4a03_4ebb_9207_95ebc38b6e84.slice/crio-7da5e2b9c1f9aaa94b21e9f15def1a5c154107ab7e75a4195cf40b9e19c85c9f WatchSource:0}: Error finding container 7da5e2b9c1f9aaa94b21e9f15def1a5c154107ab7e75a4195cf40b9e19c85c9f: Status 404 returned error can't find the container with id 7da5e2b9c1f9aaa94b21e9f15def1a5c154107ab7e75a4195cf40b9e19c85c9f
Sep 19 15:34:20.468520 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:20.468441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm" event={"ID":"32dd25e7-4a03-4ebb-9207-95ebc38b6e84","Type":"ContainerStarted","Data":"d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7"}
Sep 19 15:34:20.468520 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:20.468484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm" event={"ID":"32dd25e7-4a03-4ebb-9207-95ebc38b6e84","Type":"ContainerStarted","Data":"7da5e2b9c1f9aaa94b21e9f15def1a5c154107ab7e75a4195cf40b9e19c85c9f"}
Sep 19 15:34:20.484101 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:20.484050    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm" podStartSLOduration=2.484031357 podStartE2EDuration="2.484031357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:34:20.48319107 +0000 UTC m=+1322.376754709" watchObservedRunningTime="2026-09-19 15:34:20.484031357 +0000 UTC m=+1322.377594999"
Sep 19 15:34:29.249164 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:29.249126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:29.249536 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:29.249180    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:29.261591 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:29.261560    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:34:29.525514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:34:29.525441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:35:09.421199 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:09.421166    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"]
Sep 19 15:35:18.307485 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.307444    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"]
Sep 19 15:35:18.308075 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.307694    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm" podUID="223e26a2-8d92-4b15-a565-ede51d9530d0" containerName="payload-pre-processing" containerID="cri-o://d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e" gracePeriod=30
Sep 19 15:35:18.561690 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.561639    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:35:18.685469 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.685438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223e26a2-8d92-4b15-a565-ede51d9530d0-plugins-config-volume\") pod \"223e26a2-8d92-4b15-a565-ede51d9530d0\" (UID: \"223e26a2-8d92-4b15-a565-ede51d9530d0\") "
Sep 19 15:35:18.685647 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.685479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6brdd\" (UniqueName: \"kubernetes.io/projected/223e26a2-8d92-4b15-a565-ede51d9530d0-kube-api-access-6brdd\") pod \"223e26a2-8d92-4b15-a565-ede51d9530d0\" (UID: \"223e26a2-8d92-4b15-a565-ede51d9530d0\") "
Sep 19 15:35:18.685860 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.685835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223e26a2-8d92-4b15-a565-ede51d9530d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "223e26a2-8d92-4b15-a565-ede51d9530d0" (UID: "223e26a2-8d92-4b15-a565-ede51d9530d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:35:18.687652 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.687632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223e26a2-8d92-4b15-a565-ede51d9530d0-kube-api-access-6brdd" (OuterVolumeSpecName: "kube-api-access-6brdd") pod "223e26a2-8d92-4b15-a565-ede51d9530d0" (UID: "223e26a2-8d92-4b15-a565-ede51d9530d0"). InnerVolumeSpecName "kube-api-access-6brdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:35:18.753358 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.753330    2579 generic.go:356] "Generic (PLEG): container finished" podID="223e26a2-8d92-4b15-a565-ede51d9530d0" containerID="d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e" exitCode=0
Sep 19 15:35:18.753461 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.753378    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"
Sep 19 15:35:18.753461 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.753419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm" event={"ID":"223e26a2-8d92-4b15-a565-ede51d9530d0","Type":"ContainerDied","Data":"d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e"}
Sep 19 15:35:18.753580 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.753481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm" event={"ID":"223e26a2-8d92-4b15-a565-ede51d9530d0","Type":"ContainerDied","Data":"627cf0de40bf26381e3b29b44153cb5b58bb2cd169f8f8fd5a98d6beef469658"}
Sep 19 15:35:18.753580 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.753513    2579 scope.go:117] "RemoveContainer" containerID="d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e"
Sep 19 15:35:18.762562 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.762545    2579 scope.go:117] "RemoveContainer" containerID="d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e"
Sep 19 15:35:18.762891 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:35:18.762872    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e\": container with ID starting with d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e not found: ID does not exist" containerID="d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e"
Sep 19 15:35:18.762950 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.762898    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e"} err="failed to get container status \"d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e\": rpc error: code = NotFound desc = could not find container \"d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e\": container with ID starting with d5a61ef7e5cc1c34c21136baa7eabb72dd9f7effe7a4a3f2acac4ed15b15780e not found: ID does not exist"
Sep 19 15:35:18.773210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.773187    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"]
Sep 19 15:35:18.776952 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.776931    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21d2a4-6f77bb7c58-lwjkm"]
Sep 19 15:35:18.786393 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.786373    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223e26a2-8d92-4b15-a565-ede51d9530d0-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:18.786453 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:18.786398    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6brdd\" (UniqueName: \"kubernetes.io/projected/223e26a2-8d92-4b15-a565-ede51d9530d0-kube-api-access-6brdd\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:20.170935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:20.170901    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223e26a2-8d92-4b15-a565-ede51d9530d0" path="/var/lib/kubelet/pods/223e26a2-8d92-4b15-a565-ede51d9530d0/volumes"
Sep 19 15:35:24.440665 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.440626    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d" podUID="3e621478-2084-4944-994d-e84aa84c9a9e" containerName="main" containerID="cri-o://286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426" gracePeriod=45
Sep 19 15:35:24.691965 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.691906    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:35:24.783939 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.783906    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e621478-2084-4944-994d-e84aa84c9a9e" containerID="286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426" exitCode=0
Sep 19 15:35:24.784061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.783945    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"
Sep 19 15:35:24.784061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.783987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d" event={"ID":"3e621478-2084-4944-994d-e84aa84c9a9e","Type":"ContainerDied","Data":"286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426"}
Sep 19 15:35:24.784061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.784027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d" event={"ID":"3e621478-2084-4944-994d-e84aa84c9a9e","Type":"ContainerDied","Data":"ddd44a0470bde7a78ec584aa612faffe9748aba016c18c569ffe21ac061b199a"}
Sep 19 15:35:24.784061 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.784051    2579 scope.go:117] "RemoveContainer" containerID="286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426"
Sep 19 15:35:24.792149 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.792132    2579 scope.go:117] "RemoveContainer" containerID="286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426"
Sep 19 15:35:24.792398 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:35:24.792375    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426\": container with ID starting with 286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426 not found: ID does not exist" containerID="286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426"
Sep 19 15:35:24.792453 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.792410    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426"} err="failed to get container status \"286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426\": rpc error: code = NotFound desc = could not find container \"286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426\": container with ID starting with 286eea8a9ace9981b1707adb4f1236fb6c44040f4cd27e4a7eacfe5ada0b3426 not found: ID does not exist"
Sep 19 15:35:24.836314 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-tmp-dir\") pod \"3e621478-2084-4944-994d-e84aa84c9a9e\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") "
Sep 19 15:35:24.836406 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkphd\" (UniqueName: \"kubernetes.io/projected/3e621478-2084-4944-994d-e84aa84c9a9e-kube-api-access-lkphd\") pod \"3e621478-2084-4944-994d-e84aa84c9a9e\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") "
Sep 19 15:35:24.836406 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-dshm\") pod \"3e621478-2084-4944-994d-e84aa84c9a9e\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") "
Sep 19 15:35:24.836406 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836393    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-home\") pod \"3e621478-2084-4944-994d-e84aa84c9a9e\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") "
Sep 19 15:35:24.836524 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-model-cache\") pod \"3e621478-2084-4944-994d-e84aa84c9a9e\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") "
Sep 19 15:35:24.836524 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836503    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e621478-2084-4944-994d-e84aa84c9a9e-tls-certs\") pod \"3e621478-2084-4944-994d-e84aa84c9a9e\" (UID: \"3e621478-2084-4944-994d-e84aa84c9a9e\") "
Sep 19 15:35:24.836648 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e621478-2084-4944-994d-e84aa84c9a9e" (UID: "3e621478-2084-4944-994d-e84aa84c9a9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:35:24.836711 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-home" (OuterVolumeSpecName: "home") pod "3e621478-2084-4944-994d-e84aa84c9a9e" (UID: "3e621478-2084-4944-994d-e84aa84c9a9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:35:24.836711 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e621478-2084-4944-994d-e84aa84c9a9e" (UID: "3e621478-2084-4944-994d-e84aa84c9a9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:35:24.836880 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836864    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:24.836928 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836882    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:24.836928 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.836889    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:24.838704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.838659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-dshm" (OuterVolumeSpecName: "dshm") pod "3e621478-2084-4944-994d-e84aa84c9a9e" (UID: "3e621478-2084-4944-994d-e84aa84c9a9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:35:24.838704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.838692    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e621478-2084-4944-994d-e84aa84c9a9e-kube-api-access-lkphd" (OuterVolumeSpecName: "kube-api-access-lkphd") pod "3e621478-2084-4944-994d-e84aa84c9a9e" (UID: "3e621478-2084-4944-994d-e84aa84c9a9e"). InnerVolumeSpecName "kube-api-access-lkphd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:35:24.838848 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.838785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e621478-2084-4944-994d-e84aa84c9a9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e621478-2084-4944-994d-e84aa84c9a9e" (UID: "3e621478-2084-4944-994d-e84aa84c9a9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:35:24.938219 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.938191    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e621478-2084-4944-994d-e84aa84c9a9e-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:24.938219 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.938219    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkphd\" (UniqueName: \"kubernetes.io/projected/3e621478-2084-4944-994d-e84aa84c9a9e-kube-api-access-lkphd\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:24.938338 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:24.938229    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e621478-2084-4944-994d-e84aa84c9a9e-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:25.106800 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.106755    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"]
Sep 19 15:35:25.110434 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.110413    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e93f57/test-model-e93f57-kserve-7b9fffdd59-k828d"]
Sep 19 15:35:25.532467 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.532436    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"]
Sep 19 15:35:25.532854 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.532839    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e621478-2084-4944-994d-e84aa84c9a9e" containerName="main"
Sep 19 15:35:25.532893 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.532856    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e621478-2084-4944-994d-e84aa84c9a9e" containerName="main"
Sep 19 15:35:25.532893 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.532867    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223e26a2-8d92-4b15-a565-ede51d9530d0" containerName="payload-pre-processing"
Sep 19 15:35:25.532893 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.532873    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="223e26a2-8d92-4b15-a565-ede51d9530d0" containerName="payload-pre-processing"
Sep 19 15:35:25.532987 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.532939    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="223e26a2-8d92-4b15-a565-ede51d9530d0" containerName="payload-pre-processing"
Sep 19 15:35:25.532987 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.532949    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e621478-2084-4944-994d-e84aa84c9a9e" containerName="main"
Sep 19 15:35:25.537251 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.537232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"
Sep 19 15:35:25.539232 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.539214    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 15:35:25.539903 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.539632    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7znsl\""
Sep 19 15:35:25.541850 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.541827    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"]
Sep 19 15:35:25.643565 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.643542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256bf3c-f112-4baf-9958-45d5b35481ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8\" (UID: \"7256bf3c-f112-4baf-9958-45d5b35481ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"
Sep 19 15:35:25.744891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.744862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256bf3c-f112-4baf-9958-45d5b35481ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8\" (UID: \"7256bf3c-f112-4baf-9958-45d5b35481ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"
Sep 19 15:35:25.745417 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.745393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256bf3c-f112-4baf-9958-45d5b35481ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8\" (UID: \"7256bf3c-f112-4baf-9958-45d5b35481ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"
Sep 19 15:35:25.848187 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:25.848124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"
Sep 19 15:35:26.117483 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:26.117460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"]
Sep 19 15:35:26.119529 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:35:26.119505    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7256bf3c_f112_4baf_9958_45d5b35481ae.slice/crio-470dc7831d988d73535ba8ae07b859047402781f1d215c6783be763df7b61a84 WatchSource:0}: Error finding container 470dc7831d988d73535ba8ae07b859047402781f1d215c6783be763df7b61a84: Status 404 returned error can't find the container with id 470dc7831d988d73535ba8ae07b859047402781f1d215c6783be763df7b61a84
Sep 19 15:35:26.174444 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:26.174419    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e621478-2084-4944-994d-e84aa84c9a9e" path="/var/lib/kubelet/pods/3e621478-2084-4944-994d-e84aa84c9a9e/volumes"
Sep 19 15:35:26.796146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:26.796114    2579 generic.go:356] "Generic (PLEG): container finished" podID="7256bf3c-f112-4baf-9958-45d5b35481ae" containerID="3b02a298cd20ea29c09e23362393846cc5d21bca41797d2ada7cf3dfde829652" exitCode=0
Sep 19 15:35:26.796455 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:26.796169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8" event={"ID":"7256bf3c-f112-4baf-9958-45d5b35481ae","Type":"ContainerDied","Data":"3b02a298cd20ea29c09e23362393846cc5d21bca41797d2ada7cf3dfde829652"}
Sep 19 15:35:26.796455 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:26.796217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8" event={"ID":"7256bf3c-f112-4baf-9958-45d5b35481ae","Type":"ContainerStarted","Data":"470dc7831d988d73535ba8ae07b859047402781f1d215c6783be763df7b61a84"}
Sep 19 15:35:27.924658 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:27.924630    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"
Sep 19 15:35:28.063914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:28.063851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256bf3c-f112-4baf-9958-45d5b35481ae-maas-api-service-ca\") pod \"7256bf3c-f112-4baf-9958-45d5b35481ae\" (UID: \"7256bf3c-f112-4baf-9958-45d5b35481ae\") "
Sep 19 15:35:28.064181 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:28.064160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7256bf3c-f112-4baf-9958-45d5b35481ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7256bf3c-f112-4baf-9958-45d5b35481ae" (UID: "7256bf3c-f112-4baf-9958-45d5b35481ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:35:28.165181 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:28.165150    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7256bf3c-f112-4baf-9958-45d5b35481ae-maas-api-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:28.807012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:28.806983    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"
Sep 19 15:35:28.807167 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:28.806974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8" event={"ID":"7256bf3c-f112-4baf-9958-45d5b35481ae","Type":"ContainerDied","Data":"470dc7831d988d73535ba8ae07b859047402781f1d215c6783be763df7b61a84"}
Sep 19 15:35:28.807167 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:28.807112    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="470dc7831d988d73535ba8ae07b859047402781f1d215c6783be763df7b61a84"
Sep 19 15:35:36.875704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:36.875665    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"]
Sep 19 15:35:36.876214 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:36.875900    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t" podUID="ac1a75b2-7ff2-43f4-86e8-aea0d0933acb" containerName="payload-processing" containerID="cri-o://aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1" gracePeriod=30
Sep 19 15:35:36.881964 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:36.881924    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"]
Sep 19 15:35:36.882189 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:36.882147    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm" podUID="4c521541-0eb0-4bc8-9a6d-da6832616a0e" containerName="payload-pre-processing" containerID="cri-o://60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f" gracePeriod=30
Sep 19 15:35:37.050252 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.050219    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"]
Sep 19 15:35:37.053560 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.053529    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e93f57-nr7z8"]
Sep 19 15:35:37.121026 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.121005    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:35:37.152660 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.152636    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:35:37.237894 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.237868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-plugins-config-volume\") pod \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\" (UID: \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\") "
Sep 19 15:35:37.238045 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.237938    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw992\" (UniqueName: \"kubernetes.io/projected/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-kube-api-access-rw992\") pod \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\" (UID: \"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb\") "
Sep 19 15:35:37.238194 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.238169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac1a75b2-7ff2-43f4-86e8-aea0d0933acb" (UID: "ac1a75b2-7ff2-43f4-86e8-aea0d0933acb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:35:37.240076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.240053    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-kube-api-access-rw992" (OuterVolumeSpecName: "kube-api-access-rw992") pod "ac1a75b2-7ff2-43f4-86e8-aea0d0933acb" (UID: "ac1a75b2-7ff2-43f4-86e8-aea0d0933acb"). InnerVolumeSpecName "kube-api-access-rw992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:35:37.339374 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.339344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks9t7\" (UniqueName: \"kubernetes.io/projected/4c521541-0eb0-4bc8-9a6d-da6832616a0e-kube-api-access-ks9t7\") pod \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\" (UID: \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\") "
Sep 19 15:35:37.339514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.339430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c521541-0eb0-4bc8-9a6d-da6832616a0e-plugins-config-volume\") pod \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\" (UID: \"4c521541-0eb0-4bc8-9a6d-da6832616a0e\") "
Sep 19 15:35:37.339717 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.339702    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:37.339791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.339722    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw992\" (UniqueName: \"kubernetes.io/projected/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb-kube-api-access-rw992\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:37.339791 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.339762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c521541-0eb0-4bc8-9a6d-da6832616a0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c521541-0eb0-4bc8-9a6d-da6832616a0e" (UID: "4c521541-0eb0-4bc8-9a6d-da6832616a0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:35:37.341352 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.341331    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c521541-0eb0-4bc8-9a6d-da6832616a0e-kube-api-access-ks9t7" (OuterVolumeSpecName: "kube-api-access-ks9t7") pod "4c521541-0eb0-4bc8-9a6d-da6832616a0e" (UID: "4c521541-0eb0-4bc8-9a6d-da6832616a0e"). InnerVolumeSpecName "kube-api-access-ks9t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:35:37.441012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.440986    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c521541-0eb0-4bc8-9a6d-da6832616a0e-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:37.441105 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.441014    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks9t7\" (UniqueName: \"kubernetes.io/projected/4c521541-0eb0-4bc8-9a6d-da6832616a0e-kube-api-access-ks9t7\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:35:37.852835 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.852737    2579 generic.go:356] "Generic (PLEG): container finished" podID="ac1a75b2-7ff2-43f4-86e8-aea0d0933acb" containerID="aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1" exitCode=0
Sep 19 15:35:37.852835 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.852778    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"
Sep 19 15:35:37.852835 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.852823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t" event={"ID":"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb","Type":"ContainerDied","Data":"aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1"}
Sep 19 15:35:37.853099 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.852869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t" event={"ID":"ac1a75b2-7ff2-43f4-86e8-aea0d0933acb","Type":"ContainerDied","Data":"eb0893cdfc5a82160b89b2be2c31d04797f7c211c8e30b3ef2e5fa6a45dadd91"}
Sep 19 15:35:37.853099 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.852893    2579 scope.go:117] "RemoveContainer" containerID="aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1"
Sep 19 15:35:37.854224 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.854205    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c521541-0eb0-4bc8-9a6d-da6832616a0e" containerID="60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f" exitCode=0
Sep 19 15:35:37.854341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.854236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm" event={"ID":"4c521541-0eb0-4bc8-9a6d-da6832616a0e","Type":"ContainerDied","Data":"60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f"}
Sep 19 15:35:37.854341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.854292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm" event={"ID":"4c521541-0eb0-4bc8-9a6d-da6832616a0e","Type":"ContainerDied","Data":"d2922abc3510f9b7faf6619c2c3d3acbf1ce9700876d2a9ab29f295980a406ff"}
Sep 19 15:35:37.854341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.854246    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"
Sep 19 15:35:37.862153 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.862134    2579 scope.go:117] "RemoveContainer" containerID="aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1"
Sep 19 15:35:37.862474 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:35:37.862425    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1\": container with ID starting with aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1 not found: ID does not exist" containerID="aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1"
Sep 19 15:35:37.862474 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.862452    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1"} err="failed to get container status \"aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1\": rpc error: code = NotFound desc = could not find container \"aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1\": container with ID starting with aca3fdbab44e9b78ccd58a7fee8f1567145b76829a45675a875adb522b9a10b1 not found: ID does not exist"
Sep 19 15:35:37.862474 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.862468    2579 scope.go:117] "RemoveContainer" containerID="60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f"
Sep 19 15:35:37.870800 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.870737    2579 scope.go:117] "RemoveContainer" containerID="60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f"
Sep 19 15:35:37.871336 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:35:37.871050    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f\": container with ID starting with 60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f not found: ID does not exist" containerID="60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f"
Sep 19 15:35:37.871336 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.871086    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f"} err="failed to get container status \"60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f\": rpc error: code = NotFound desc = could not find container \"60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f\": container with ID starting with 60761abc275cd7d9a9c24a8b4e9ab4c8ec14bfd1cbc5cf9708591dc57f095a7f not found: ID does not exist"
Sep 19 15:35:37.873202 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.873181    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"]
Sep 19 15:35:37.880048 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.880031    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e93f57-5fff958f57-bkg7t"]
Sep 19 15:35:37.888916 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.888894    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"]
Sep 19 15:35:37.892058 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:37.892039    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e93f57-85f4b9d55d-jfmqm"]
Sep 19 15:35:38.170747 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:38.170679    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c521541-0eb0-4bc8-9a6d-da6832616a0e" path="/var/lib/kubelet/pods/4c521541-0eb0-4bc8-9a6d-da6832616a0e/volumes"
Sep 19 15:35:38.171016 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:38.171002    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7256bf3c-f112-4baf-9958-45d5b35481ae" path="/var/lib/kubelet/pods/7256bf3c-f112-4baf-9958-45d5b35481ae/volumes"
Sep 19 15:35:38.171319 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:35:38.171307    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac1a75b2-7ff2-43f4-86e8-aea0d0933acb" path="/var/lib/kubelet/pods/ac1a75b2-7ff2-43f4-86e8-aea0d0933acb/volumes"
Sep 19 15:36:10.532269 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532235    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"]
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532561    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7256bf3c-f112-4baf-9958-45d5b35481ae" containerName="revoke-keys"
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7256bf3c-f112-4baf-9958-45d5b35481ae" containerName="revoke-keys"
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532583    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1a75b2-7ff2-43f4-86e8-aea0d0933acb" containerName="payload-processing"
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532589    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1a75b2-7ff2-43f4-86e8-aea0d0933acb" containerName="payload-processing"
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532618    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c521541-0eb0-4bc8-9a6d-da6832616a0e" containerName="payload-pre-processing"
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532624    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c521541-0eb0-4bc8-9a6d-da6832616a0e" containerName="payload-pre-processing"
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532688    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7256bf3c-f112-4baf-9958-45d5b35481ae" containerName="revoke-keys"
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532695    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c521541-0eb0-4bc8-9a6d-da6832616a0e" containerName="payload-pre-processing"
Sep 19 15:36:10.532722 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.532704    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1a75b2-7ff2-43f4-86e8-aea0d0933acb" containerName="payload-processing"
Sep 19 15:36:10.535659 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.535641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:10.537914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.537894    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-c8e1cf-dockercfg-bb8hj\""
Sep 19 15:36:10.537914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.537907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-c8e1cf\""
Sep 19 15:36:10.547067 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.547048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"]
Sep 19 15:36:10.560689 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.560660    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"]
Sep 19 15:36:10.564137 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.564121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:10.576267 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.576246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"]
Sep 19 15:36:10.602091 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.602069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj\" (UID: \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:10.602187 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.602128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxfwc\" (UniqueName: \"kubernetes.io/projected/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-kube-api-access-jxfwc\") pod \"payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj\" (UID: \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:10.703019 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.702993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9g6l\" (UniqueName: \"kubernetes.io/projected/cba2ceb3-0102-467e-8633-3c8a005a2bb4-kube-api-access-p9g6l\") pod \"payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4\" (UID: \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:10.703124 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.703035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba2ceb3-0102-467e-8633-3c8a005a2bb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4\" (UID: \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:10.703124 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.703062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj\" (UID: \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:10.703204 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.703127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxfwc\" (UniqueName: \"kubernetes.io/projected/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-kube-api-access-jxfwc\") pod \"payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj\" (UID: \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:10.703677 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.703661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj\" (UID: \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:10.710998 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.710978    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxfwc\" (UniqueName: \"kubernetes.io/projected/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-kube-api-access-jxfwc\") pod \"payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj\" (UID: \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:10.804523 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.804464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba2ceb3-0102-467e-8633-3c8a005a2bb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4\" (UID: \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:10.804710 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.804536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9g6l\" (UniqueName: \"kubernetes.io/projected/cba2ceb3-0102-467e-8633-3c8a005a2bb4-kube-api-access-p9g6l\") pod \"payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4\" (UID: \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:10.805149 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.805131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba2ceb3-0102-467e-8633-3c8a005a2bb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4\" (UID: \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:10.812284 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.812261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9g6l\" (UniqueName: \"kubernetes.io/projected/cba2ceb3-0102-467e-8633-3c8a005a2bb4-kube-api-access-p9g6l\") pod \"payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4\" (UID: \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:10.845742 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.845714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:10.874435 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:10.874402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:11.113217 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:11.113146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"]
Sep 19 15:36:11.115430 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:36:11.115406    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ffa437_b79c_4f4f_a4ff_f47de85cff3c.slice/crio-4fcef283c9a7d90bcb8ad7edc02a3453a14cd237bf5c92453825224e042bb370 WatchSource:0}: Error finding container 4fcef283c9a7d90bcb8ad7edc02a3453a14cd237bf5c92453825224e042bb370: Status 404 returned error can't find the container with id 4fcef283c9a7d90bcb8ad7edc02a3453a14cd237bf5c92453825224e042bb370
Sep 19 15:36:11.149338 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:11.149317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"]
Sep 19 15:36:11.151155 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:36:11.151130    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba2ceb3_0102_467e_8633_3c8a005a2bb4.slice/crio-444bf4891430a713f72e5f8db9b41042bb229c7819a527808008886010cfc98c WatchSource:0}: Error finding container 444bf4891430a713f72e5f8db9b41042bb229c7819a527808008886010cfc98c: Status 404 returned error can't find the container with id 444bf4891430a713f72e5f8db9b41042bb229c7819a527808008886010cfc98c
Sep 19 15:36:12.023063 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:12.023030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4" event={"ID":"cba2ceb3-0102-467e-8633-3c8a005a2bb4","Type":"ContainerStarted","Data":"032376d85c414ed19a1aa80bf4c801394f243e822ac06225dcffb24b96d1c2d6"}
Sep 19 15:36:12.023514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:12.023083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4" event={"ID":"cba2ceb3-0102-467e-8633-3c8a005a2bb4","Type":"ContainerStarted","Data":"444bf4891430a713f72e5f8db9b41042bb229c7819a527808008886010cfc98c"}
Sep 19 15:36:12.023514 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:12.023136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:12.024699 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:12.024676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj" event={"ID":"13ffa437-b79c-4f4f-a4ff-f47de85cff3c","Type":"ContainerStarted","Data":"80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc"}
Sep 19 15:36:12.024810 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:12.024712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj" event={"ID":"13ffa437-b79c-4f4f-a4ff-f47de85cff3c","Type":"ContainerStarted","Data":"4fcef283c9a7d90bcb8ad7edc02a3453a14cd237bf5c92453825224e042bb370"}
Sep 19 15:36:12.024810 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:12.024741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:12.041096 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:12.041048    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4" podStartSLOduration=2.041034677 podStartE2EDuration="2.041034677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:36:12.039629633 +0000 UTC m=+1433.933193285" watchObservedRunningTime="2026-09-19 15:36:12.041034677 +0000 UTC m=+1433.934598316"
Sep 19 15:36:12.055161 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:12.055109    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj" podStartSLOduration=2.055094253 podStartE2EDuration="2.055094253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:36:12.053202627 +0000 UTC m=+1433.946766267" watchObservedRunningTime="2026-09-19 15:36:12.055094253 +0000 UTC m=+1433.948657894"
Sep 19 15:36:21.471672 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.471633    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"]
Sep 19 15:36:21.476422 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.476400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:21.478381 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.478355    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f9c5c2\""
Sep 19 15:36:21.486751 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.486731    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"]
Sep 19 15:36:21.546909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.546877    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"]
Sep 19 15:36:21.552027 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.552002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:21.554055 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.554036    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f9c5c2-dockercfg-wblkq\""
Sep 19 15:36:21.558718 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.558697    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"]
Sep 19 15:36:21.595245 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.595219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55bs\" (UniqueName: \"kubernetes.io/projected/0e6e577a-133c-4b7a-80c7-d2072c826c5e-kube-api-access-w55bs\") pod \"payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp\" (UID: \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:21.595382 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.595342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6e577a-133c-4b7a-80c7-d2072c826c5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp\" (UID: \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:21.696160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.696131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6e577a-133c-4b7a-80c7-d2072c826c5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp\" (UID: \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:21.696291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.696173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a5468b-22ad-4499-b097-cce315e3a397-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp\" (UID: \"08a5468b-22ad-4499-b097-cce315e3a397\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:21.696291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.696202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55bs\" (UniqueName: \"kubernetes.io/projected/0e6e577a-133c-4b7a-80c7-d2072c826c5e-kube-api-access-w55bs\") pod \"payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp\" (UID: \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:21.696291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.696233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqm6\" (UniqueName: \"kubernetes.io/projected/08a5468b-22ad-4499-b097-cce315e3a397-kube-api-access-6gqm6\") pod \"payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp\" (UID: \"08a5468b-22ad-4499-b097-cce315e3a397\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:21.696841 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.696823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6e577a-133c-4b7a-80c7-d2072c826c5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp\" (UID: \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:21.705511 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.705487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55bs\" (UniqueName: \"kubernetes.io/projected/0e6e577a-133c-4b7a-80c7-d2072c826c5e-kube-api-access-w55bs\") pod \"payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp\" (UID: \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:21.788784 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.788714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:21.797367 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.797341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a5468b-22ad-4499-b097-cce315e3a397-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp\" (UID: \"08a5468b-22ad-4499-b097-cce315e3a397\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:21.797442 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.797428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqm6\" (UniqueName: \"kubernetes.io/projected/08a5468b-22ad-4499-b097-cce315e3a397-kube-api-access-6gqm6\") pod \"payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp\" (UID: \"08a5468b-22ad-4499-b097-cce315e3a397\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:21.798086 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.797992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a5468b-22ad-4499-b097-cce315e3a397-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp\" (UID: \"08a5468b-22ad-4499-b097-cce315e3a397\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:21.804628 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.804580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqm6\" (UniqueName: \"kubernetes.io/projected/08a5468b-22ad-4499-b097-cce315e3a397-kube-api-access-6gqm6\") pod \"payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp\" (UID: \"08a5468b-22ad-4499-b097-cce315e3a397\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:21.864114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:21.864086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:22.056643 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:22.056619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"]
Sep 19 15:36:22.058492 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:36:22.058466    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6e577a_133c_4b7a_80c7_d2072c826c5e.slice/crio-5a7cec78526c205552c9f16429c8ab2f5e1e7b90982c6206737603f7df1ca0f3 WatchSource:0}: Error finding container 5a7cec78526c205552c9f16429c8ab2f5e1e7b90982c6206737603f7df1ca0f3: Status 404 returned error can't find the container with id 5a7cec78526c205552c9f16429c8ab2f5e1e7b90982c6206737603f7df1ca0f3
Sep 19 15:36:22.074103 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:22.074079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp" event={"ID":"0e6e577a-133c-4b7a-80c7-d2072c826c5e","Type":"ContainerStarted","Data":"5a7cec78526c205552c9f16429c8ab2f5e1e7b90982c6206737603f7df1ca0f3"}
Sep 19 15:36:22.142168 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:22.142102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"]
Sep 19 15:36:22.144258 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:36:22.144222    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a5468b_22ad_4499_b097_cce315e3a397.slice/crio-f0eac800460bf350bbf76a643a33ad0cae8cb67c20436764a126e7dcb4034850 WatchSource:0}: Error finding container f0eac800460bf350bbf76a643a33ad0cae8cb67c20436764a126e7dcb4034850: Status 404 returned error can't find the container with id f0eac800460bf350bbf76a643a33ad0cae8cb67c20436764a126e7dcb4034850
Sep 19 15:36:23.030786 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.030754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:36:23.031242 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.030831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:36:23.083318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.083280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp" event={"ID":"08a5468b-22ad-4499-b097-cce315e3a397","Type":"ContainerStarted","Data":"f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f"}
Sep 19 15:36:23.083516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.083437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:23.083516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.083476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp" event={"ID":"08a5468b-22ad-4499-b097-cce315e3a397","Type":"ContainerStarted","Data":"f0eac800460bf350bbf76a643a33ad0cae8cb67c20436764a126e7dcb4034850"}
Sep 19 15:36:23.085576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.085551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp" event={"ID":"0e6e577a-133c-4b7a-80c7-d2072c826c5e","Type":"ContainerStarted","Data":"87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e"}
Sep 19 15:36:23.085998 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.085979    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:23.097649 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.097591    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp" podStartSLOduration=2.097578198 podStartE2EDuration="2.097578198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:36:23.095465995 +0000 UTC m=+1444.989029636" watchObservedRunningTime="2026-09-19 15:36:23.097578198 +0000 UTC m=+1444.991141839"
Sep 19 15:36:23.114214 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:23.114175    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp" podStartSLOduration=2.114162891 podStartE2EDuration="2.114162891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:36:23.11267082 +0000 UTC m=+1445.006234461" watchObservedRunningTime="2026-09-19 15:36:23.114162891 +0000 UTC m=+1445.007726530"
Sep 19 15:36:26.993853 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:26.993820    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"]
Sep 19 15:36:26.997414 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:26.997398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:26.999318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:26.999296    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-da587f\""
Sep 19 15:36:26.999418 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:26.999363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-da587f-dockercfg-xrbfx\""
Sep 19 15:36:27.008861 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.008840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"]
Sep 19 15:36:27.030835 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.030809    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"]
Sep 19 15:36:27.034110 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.034094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:27.043952 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.043931    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"]
Sep 19 15:36:27.147159 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.147124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq\" (UID: \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:27.147289 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.147217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r426k\" (UniqueName: \"kubernetes.io/projected/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-kube-api-access-r426k\") pod \"payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf\" (UID: \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:27.147289 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.147247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf\" (UID: \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:27.147289 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.147272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xkn5\" (UniqueName: \"kubernetes.io/projected/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-kube-api-access-2xkn5\") pod \"payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq\" (UID: \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:27.248069 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.248001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq\" (UID: \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:27.248069 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.248040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r426k\" (UniqueName: \"kubernetes.io/projected/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-kube-api-access-r426k\") pod \"payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf\" (UID: \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:27.248069 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.248059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf\" (UID: \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:27.248245 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.248094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xkn5\" (UniqueName: \"kubernetes.io/projected/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-kube-api-access-2xkn5\") pod \"payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq\" (UID: \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:27.248919 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.248898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf\" (UID: \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:27.248973 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.248898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq\" (UID: \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:27.255052 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.255028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r426k\" (UniqueName: \"kubernetes.io/projected/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-kube-api-access-r426k\") pod \"payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf\" (UID: \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:27.255140 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.255049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xkn5\" (UniqueName: \"kubernetes.io/projected/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-kube-api-access-2xkn5\") pod \"payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq\" (UID: \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:27.309001 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.308978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:27.344084 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.344056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:27.579464 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.579435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"]
Sep 19 15:36:27.581421 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:36:27.581399    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d13c49b_6b40_42bd_8a6a_91ef82f961c7.slice/crio-1ed465aab774d1fb67af9b851e15d99a79b120e1946fde4e83a7209bf2a51974 WatchSource:0}: Error finding container 1ed465aab774d1fb67af9b851e15d99a79b120e1946fde4e83a7209bf2a51974: Status 404 returned error can't find the container with id 1ed465aab774d1fb67af9b851e15d99a79b120e1946fde4e83a7209bf2a51974
Sep 19 15:36:27.622154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:27.622122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"]
Sep 19 15:36:27.623102 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:36:27.623079    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918c32a2_a1a7_43d6_b59c_72c3c4a26fd0.slice/crio-76cc9f30c160a7dedc0b0b923e79812f89a7326782635a93a8dc42cfe1ec6009 WatchSource:0}: Error finding container 76cc9f30c160a7dedc0b0b923e79812f89a7326782635a93a8dc42cfe1ec6009: Status 404 returned error can't find the container with id 76cc9f30c160a7dedc0b0b923e79812f89a7326782635a93a8dc42cfe1ec6009
Sep 19 15:36:28.111657 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:28.111615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf" event={"ID":"8d13c49b-6b40-42bd-8a6a-91ef82f961c7","Type":"ContainerStarted","Data":"dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660"}
Sep 19 15:36:28.112036 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:28.111677    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:28.112036 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:28.111699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf" event={"ID":"8d13c49b-6b40-42bd-8a6a-91ef82f961c7","Type":"ContainerStarted","Data":"1ed465aab774d1fb67af9b851e15d99a79b120e1946fde4e83a7209bf2a51974"}
Sep 19 15:36:28.113174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:28.113154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq" event={"ID":"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0","Type":"ContainerStarted","Data":"e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1"}
Sep 19 15:36:28.113222 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:28.113191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq" event={"ID":"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0","Type":"ContainerStarted","Data":"76cc9f30c160a7dedc0b0b923e79812f89a7326782635a93a8dc42cfe1ec6009"}
Sep 19 15:36:28.113222 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:28.113211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:28.125899 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:28.125850    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf" podStartSLOduration=2.125832093 podStartE2EDuration="2.125832093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:36:28.124623545 +0000 UTC m=+1450.018187176" watchObservedRunningTime="2026-09-19 15:36:28.125832093 +0000 UTC m=+1450.019395734"
Sep 19 15:36:28.140186 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:28.140145    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq" podStartSLOduration=1.140131898 podStartE2EDuration="1.140131898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:36:28.138438106 +0000 UTC m=+1450.032001747" watchObservedRunningTime="2026-09-19 15:36:28.140131898 +0000 UTC m=+1450.033695548"
Sep 19 15:36:34.091891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:34.091860    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:36:34.092263 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:34.092206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:36:39.121481 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:39.121447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:36:39.122054 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:39.121525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:36:58.354165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.354130    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"]
Sep 19 15:36:58.359040 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.359013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:58.361213 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.361190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7cbcece2\""
Sep 19 15:36:58.361325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.361260    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7cbcece2-dockercfg-zkll7\""
Sep 19 15:36:58.368191 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.368167    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"]
Sep 19 15:36:58.403350 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.403324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sk2q\" (UniqueName: \"kubernetes.io/projected/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-kube-api-access-6sk2q\") pod \"payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2\" (UID: \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\") " pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:58.403447 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.403354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-plugins-config-volume\") pod \"payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2\" (UID: \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\") " pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:58.504497 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.504470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sk2q\" (UniqueName: \"kubernetes.io/projected/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-kube-api-access-6sk2q\") pod \"payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2\" (UID: \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\") " pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:58.504629 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.504502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-plugins-config-volume\") pod \"payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2\" (UID: \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\") " pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:58.505182 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.505164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-plugins-config-volume\") pod \"payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2\" (UID: \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\") " pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:58.512405 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.512388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sk2q\" (UniqueName: \"kubernetes.io/projected/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-kube-api-access-6sk2q\") pod \"payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2\" (UID: \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\") " pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:58.672395 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.672338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:58.941511 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:58.941487    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"]
Sep 19 15:36:58.943287 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:36:58.943260    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e6ddcf_2f9a_44ea_b4cc_c7ccc9b86b5e.slice/crio-cb29508d537a6f0f5e1dcff06abe417045a15f7c95cf3367fb390e8488fba2ea WatchSource:0}: Error finding container cb29508d537a6f0f5e1dcff06abe417045a15f7c95cf3367fb390e8488fba2ea: Status 404 returned error can't find the container with id cb29508d537a6f0f5e1dcff06abe417045a15f7c95cf3367fb390e8488fba2ea
Sep 19 15:36:59.276303 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:59.276219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2" event={"ID":"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e","Type":"ContainerStarted","Data":"1e371550f8a427673a6f7406d6f0099966acfb315dbebebc6923bbb46c625d25"}
Sep 19 15:36:59.276440 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:59.276362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:36:59.276440 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:59.276387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2" event={"ID":"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e","Type":"ContainerStarted","Data":"cb29508d537a6f0f5e1dcff06abe417045a15f7c95cf3367fb390e8488fba2ea"}
Sep 19 15:36:59.293754 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:36:59.293709    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2" podStartSLOduration=1.293691002 podStartE2EDuration="1.293691002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:36:59.292721979 +0000 UTC m=+1481.186285618" watchObservedRunningTime="2026-09-19 15:36:59.293691002 +0000 UTC m=+1481.187254645"
Sep 19 15:37:10.282460 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:10.282432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:37:20.193811 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:20.193786    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:37:20.196463 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:20.196445    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:37:20.263154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:20.263116    2579 scope.go:117] "RemoveContainer" containerID="742ed2502ca0b745978c9333e672212b0332d7c54ff44998f08b201608ddee0e"
Sep 19 15:37:31.561838 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:31.561735    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"]
Sep 19 15:37:31.588642 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:31.588611    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"]
Sep 19 15:37:39.939301 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:39.939268    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"]
Sep 19 15:37:39.942846 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:39.942827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:39.945577 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:39.945553    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7cbcece2\"/\"e2e-lifecycle-model-7cbcece2-kserve-self-signed-certs\""
Sep 19 15:37:39.945577 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:39.945567    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7cbcece2\"/\"default-dockercfg-jcln5\""
Sep 19 15:37:39.945746 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:39.945579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7cbcece2\"/\"kube-root-ca.crt\""
Sep 19 15:37:39.945746 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:39.945585    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7cbcece2\"/\"openshift-service-ca.crt\""
Sep 19 15:37:39.950456 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:39.950437    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"]
Sep 19 15:37:40.044935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.044910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-model-cache\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.045029 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.044953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea9189a-18dc-443d-ad45-1030bc5f5271-tls-certs\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.045029 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.044997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-tmp-dir\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.045107 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.045066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-dshm\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.045107 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.045088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-home\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.045211 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.045189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqs6\" (UniqueName: \"kubernetes.io/projected/3ea9189a-18dc-443d-ad45-1030bc5f5271-kube-api-access-gjqs6\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.145951 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.145924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-dshm\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.146052 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.145959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-home\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.146111 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.146075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqs6\" (UniqueName: \"kubernetes.io/projected/3ea9189a-18dc-443d-ad45-1030bc5f5271-kube-api-access-gjqs6\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.146111 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.146106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-model-cache\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.146214 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.146150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea9189a-18dc-443d-ad45-1030bc5f5271-tls-certs\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.146214 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.146167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-tmp-dir\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.146388 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.146368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-home\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.146489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.146468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-model-cache\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.146536 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.146502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-tmp-dir\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.148240 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.148219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-dshm\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.148575 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.148559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea9189a-18dc-443d-ad45-1030bc5f5271-tls-certs\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.154093 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.154074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqs6\" (UniqueName: \"kubernetes.io/projected/3ea9189a-18dc-443d-ad45-1030bc5f5271-kube-api-access-gjqs6\") pod \"e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") " pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.252809 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.252764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:40.523259 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:40.523233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"]
Sep 19 15:37:40.524864 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:37:40.524829    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ea9189a_18dc_443d_ad45_1030bc5f5271.slice/crio-b9d15f3f98bc9e0aed9aef55c3c82a7f0169de19b2a2db11a4a284eaa7ff8d3b WatchSource:0}: Error finding container b9d15f3f98bc9e0aed9aef55c3c82a7f0169de19b2a2db11a4a284eaa7ff8d3b: Status 404 returned error can't find the container with id b9d15f3f98bc9e0aed9aef55c3c82a7f0169de19b2a2db11a4a284eaa7ff8d3b
Sep 19 15:37:41.495991 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:41.495959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl" event={"ID":"3ea9189a-18dc-443d-ad45-1030bc5f5271","Type":"ContainerStarted","Data":"2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46"}
Sep 19 15:37:41.495991 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:41.495997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl" event={"ID":"3ea9189a-18dc-443d-ad45-1030bc5f5271","Type":"ContainerStarted","Data":"b9d15f3f98bc9e0aed9aef55c3c82a7f0169de19b2a2db11a4a284eaa7ff8d3b"}
Sep 19 15:37:41.511337 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:41.511284    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl" podStartSLOduration=2.511269791 podStartE2EDuration="2.511269791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:37:41.509667193 +0000 UTC m=+1523.403230827" watchObservedRunningTime="2026-09-19 15:37:41.511269791 +0000 UTC m=+1523.404833430"
Sep 19 15:37:42.401298 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.401262    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"]
Sep 19 15:37:42.401548 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.401521    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx" podUID="79d0f7d5-2e94-4aaf-be23-85251088c4b4" containerName="payload-processing" containerID="cri-o://e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a" gracePeriod=30
Sep 19 15:37:42.410439 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.409632    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"]
Sep 19 15:37:42.410439 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.409879    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d" podUID="72f3601e-4fd4-4936-ba9f-f0a25db7d5f6" containerName="payload-pre-processing" containerID="cri-o://d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d" gracePeriod=30
Sep 19 15:37:42.692809 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.692782    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:37:42.695721 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.695705    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:37:42.869828 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.869794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-plugins-config-volume\") pod \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\" (UID: \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\") "
Sep 19 15:37:42.869992 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.869845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ktzv\" (UniqueName: \"kubernetes.io/projected/79d0f7d5-2e94-4aaf-be23-85251088c4b4-kube-api-access-7ktzv\") pod \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\" (UID: \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\") "
Sep 19 15:37:42.869992 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.869881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d0f7d5-2e94-4aaf-be23-85251088c4b4-plugins-config-volume\") pod \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\" (UID: \"79d0f7d5-2e94-4aaf-be23-85251088c4b4\") "
Sep 19 15:37:42.869992 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.869911    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrc4n\" (UniqueName: \"kubernetes.io/projected/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-kube-api-access-hrc4n\") pod \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\" (UID: \"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6\") "
Sep 19 15:37:42.870155 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.870122    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72f3601e-4fd4-4936-ba9f-f0a25db7d5f6" (UID: "72f3601e-4fd4-4936-ba9f-f0a25db7d5f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:37:42.870217 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.870200    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:42.870271 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.870218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d0f7d5-2e94-4aaf-be23-85251088c4b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79d0f7d5-2e94-4aaf-be23-85251088c4b4" (UID: "79d0f7d5-2e94-4aaf-be23-85251088c4b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:37:42.872109 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.872086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-kube-api-access-hrc4n" (OuterVolumeSpecName: "kube-api-access-hrc4n") pod "72f3601e-4fd4-4936-ba9f-f0a25db7d5f6" (UID: "72f3601e-4fd4-4936-ba9f-f0a25db7d5f6"). InnerVolumeSpecName "kube-api-access-hrc4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:37:42.872208 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.872178    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d0f7d5-2e94-4aaf-be23-85251088c4b4-kube-api-access-7ktzv" (OuterVolumeSpecName: "kube-api-access-7ktzv") pod "79d0f7d5-2e94-4aaf-be23-85251088c4b4" (UID: "79d0f7d5-2e94-4aaf-be23-85251088c4b4"). InnerVolumeSpecName "kube-api-access-7ktzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:37:42.971097 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.971073    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktzv\" (UniqueName: \"kubernetes.io/projected/79d0f7d5-2e94-4aaf-be23-85251088c4b4-kube-api-access-7ktzv\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:42.971097 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.971095    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d0f7d5-2e94-4aaf-be23-85251088c4b4-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:42.971241 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:42.971104    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrc4n\" (UniqueName: \"kubernetes.io/projected/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6-kube-api-access-hrc4n\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:43.507300 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.507270    2579 generic.go:356] "Generic (PLEG): container finished" podID="72f3601e-4fd4-4936-ba9f-f0a25db7d5f6" containerID="d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d" exitCode=0
Sep 19 15:37:43.507474 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.507316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"
Sep 19 15:37:43.507474 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.507364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d" event={"ID":"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6","Type":"ContainerDied","Data":"d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d"}
Sep 19 15:37:43.507474 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.507420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d" event={"ID":"72f3601e-4fd4-4936-ba9f-f0a25db7d5f6","Type":"ContainerDied","Data":"86ca838f6077c7007c186b8de6a06f40f208e7b1273ad8a3ec99f459239230f0"}
Sep 19 15:37:43.507474 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.507447    2579 scope.go:117] "RemoveContainer" containerID="d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d"
Sep 19 15:37:43.508844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.508797    2579 generic.go:356] "Generic (PLEG): container finished" podID="79d0f7d5-2e94-4aaf-be23-85251088c4b4" containerID="e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a" exitCode=0
Sep 19 15:37:43.508844 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.508840    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"
Sep 19 15:37:43.508963 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.508889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx" event={"ID":"79d0f7d5-2e94-4aaf-be23-85251088c4b4","Type":"ContainerDied","Data":"e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a"}
Sep 19 15:37:43.508963 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.508923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx" event={"ID":"79d0f7d5-2e94-4aaf-be23-85251088c4b4","Type":"ContainerDied","Data":"e16fe9d9977edef9503dc897137ccfa213eaa03cc6b25e23be4c5af63ad8cf21"}
Sep 19 15:37:43.517622 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.517581    2579 scope.go:117] "RemoveContainer" containerID="d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d"
Sep 19 15:37:43.517953 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:37:43.517935    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d\": container with ID starting with d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d not found: ID does not exist" containerID="d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d"
Sep 19 15:37:43.518019 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.517964    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d"} err="failed to get container status \"d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d\": rpc error: code = NotFound desc = could not find container \"d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d\": container with ID starting with d37533e3ebb0d29180b8d93f337bb1151be4cf5e35872ec55005d5656c127b6d not found: ID does not exist"
Sep 19 15:37:43.518019 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.517985    2579 scope.go:117] "RemoveContainer" containerID="e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a"
Sep 19 15:37:43.531511 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.531462    2579 scope.go:117] "RemoveContainer" containerID="e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a"
Sep 19 15:37:43.531875 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:37:43.531848    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a\": container with ID starting with e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a not found: ID does not exist" containerID="e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a"
Sep 19 15:37:43.531950 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.531884    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a"} err="failed to get container status \"e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a\": rpc error: code = NotFound desc = could not find container \"e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a\": container with ID starting with e4eb454080bdd008a23fac602d0fff36ab2a4dd06bc8498895076814b232b71a not found: ID does not exist"
Sep 19 15:37:43.534154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.534132    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"]
Sep 19 15:37:43.535928 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.535908    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-33737b-76cc56d95f-bbvrx"]
Sep 19 15:37:43.543868 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.543848    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"]
Sep 19 15:37:43.546881 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:43.546860    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-33737b-94d6b7658-h597d"]
Sep 19 15:37:44.171001 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:44.170969    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72f3601e-4fd4-4936-ba9f-f0a25db7d5f6" path="/var/lib/kubelet/pods/72f3601e-4fd4-4936-ba9f-f0a25db7d5f6/volumes"
Sep 19 15:37:44.171376 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:44.171361    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79d0f7d5-2e94-4aaf-be23-85251088c4b4" path="/var/lib/kubelet/pods/79d0f7d5-2e94-4aaf-be23-85251088c4b4/volumes"
Sep 19 15:37:46.580952 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.580899    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm" podUID="32dd25e7-4a03-4ebb-9207-95ebc38b6e84" containerName="main" containerID="cri-o://d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7" gracePeriod=45
Sep 19 15:37:46.607907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.607860    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6" podUID="4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" containerName="main" containerID="cri-o://4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277" gracePeriod=45
Sep 19 15:37:46.867704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.867682    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:37:46.870536 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.870518    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:37:46.902193 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902163    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-model-cache\") pod \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") "
Sep 19 15:37:46.902331 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tmp-dir\") pod \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") "
Sep 19 15:37:46.902331 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tls-certs\") pod \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") "
Sep 19 15:37:46.902331 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-model-cache\") pod \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") "
Sep 19 15:37:46.902518 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902347    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hqhl\" (UniqueName: \"kubernetes.io/projected/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-kube-api-access-9hqhl\") pod \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") "
Sep 19 15:37:46.902518 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-dshm\") pod \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") "
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902750    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-model-cache" (OuterVolumeSpecName: "model-cache") pod "32dd25e7-4a03-4ebb-9207-95ebc38b6e84" (UID: "32dd25e7-4a03-4ebb-9207-95ebc38b6e84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" (UID: "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-home\") pod \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") "
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.902993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k29v2\" (UniqueName: \"kubernetes.io/projected/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-kube-api-access-k29v2\") pod \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") "
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903093    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tls-certs\") pod \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") "
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903098    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-home" (OuterVolumeSpecName: "home") pod "32dd25e7-4a03-4ebb-9207-95ebc38b6e84" (UID: "32dd25e7-4a03-4ebb-9207-95ebc38b6e84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tmp-dir\") pod \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") "
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-home\") pod \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\" (UID: \"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38\") "
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903175    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-dshm\") pod \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\" (UID: \"32dd25e7-4a03-4ebb-9207-95ebc38b6e84\") "
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32dd25e7-4a03-4ebb-9207-95ebc38b6e84" (UID: "32dd25e7-4a03-4ebb-9207-95ebc38b6e84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903642    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903659    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903673    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:46.903805 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903684    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:46.904744 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903857    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-home" (OuterVolumeSpecName: "home") pod "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" (UID: "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:37:46.904744 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.903996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" (UID: "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:37:46.905242 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.905218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-dshm" (OuterVolumeSpecName: "dshm") pod "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" (UID: "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:37:46.906749 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.906722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32dd25e7-4a03-4ebb-9207-95ebc38b6e84" (UID: "32dd25e7-4a03-4ebb-9207-95ebc38b6e84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:37:46.906952 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.906924    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-dshm" (OuterVolumeSpecName: "dshm") pod "32dd25e7-4a03-4ebb-9207-95ebc38b6e84" (UID: "32dd25e7-4a03-4ebb-9207-95ebc38b6e84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:37:46.907654 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.907584    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-kube-api-access-k29v2" (OuterVolumeSpecName: "kube-api-access-k29v2") pod "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" (UID: "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38"). InnerVolumeSpecName "kube-api-access-k29v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:37:46.907973 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.907949    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" (UID: "4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:37:46.908421 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:46.908393    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-kube-api-access-9hqhl" (OuterVolumeSpecName: "kube-api-access-9hqhl") pod "32dd25e7-4a03-4ebb-9207-95ebc38b6e84" (UID: "32dd25e7-4a03-4ebb-9207-95ebc38b6e84"). InnerVolumeSpecName "kube-api-access-9hqhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:37:47.004323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.004294    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hqhl\" (UniqueName: \"kubernetes.io/projected/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-kube-api-access-9hqhl\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:47.004323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.004322    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:47.004499 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.004331    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k29v2\" (UniqueName: \"kubernetes.io/projected/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-kube-api-access-k29v2\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:47.004499 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.004340    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:47.004499 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.004348    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:47.004499 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.004355    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:47.004499 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.004362    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:47.004499 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.004369    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dd25e7-4a03-4ebb-9207-95ebc38b6e84-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:37:47.531076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.531043    2579 generic.go:356] "Generic (PLEG): container finished" podID="4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" containerID="4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277" exitCode=0
Sep 19 15:37:47.531268 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.531088    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"
Sep 19 15:37:47.531268 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.531131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6" event={"ID":"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38","Type":"ContainerDied","Data":"4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277"}
Sep 19 15:37:47.531268 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.531186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6" event={"ID":"4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38","Type":"ContainerDied","Data":"737a63c215489c08f0f43dd392d82ecbdfce4958be0d0f1be6530e1b682699c5"}
Sep 19 15:37:47.531268 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.531209    2579 scope.go:117] "RemoveContainer" containerID="4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277"
Sep 19 15:37:47.532492 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.532470    2579 generic.go:356] "Generic (PLEG): container finished" podID="32dd25e7-4a03-4ebb-9207-95ebc38b6e84" containerID="d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7" exitCode=0
Sep 19 15:37:47.532628 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.532531    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"
Sep 19 15:37:47.532628 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.532531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm" event={"ID":"32dd25e7-4a03-4ebb-9207-95ebc38b6e84","Type":"ContainerDied","Data":"d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7"}
Sep 19 15:37:47.532628 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.532570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm" event={"ID":"32dd25e7-4a03-4ebb-9207-95ebc38b6e84","Type":"ContainerDied","Data":"7da5e2b9c1f9aaa94b21e9f15def1a5c154107ab7e75a4195cf40b9e19c85c9f"}
Sep 19 15:37:47.541231 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.541214    2579 scope.go:117] "RemoveContainer" containerID="4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277"
Sep 19 15:37:47.541475 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:37:47.541455    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277\": container with ID starting with 4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277 not found: ID does not exist" containerID="4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277"
Sep 19 15:37:47.541575 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.541480    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277"} err="failed to get container status \"4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277\": rpc error: code = NotFound desc = could not find container \"4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277\": container with ID starting with 4cc2878a6b0df5ac7326bcfb9cd20b4d02b6d069699836acc4469920a18ad277 not found: ID does not exist"
Sep 19 15:37:47.541575 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.541495    2579 scope.go:117] "RemoveContainer" containerID="d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7"
Sep 19 15:37:47.549265 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.549249    2579 scope.go:117] "RemoveContainer" containerID="d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7"
Sep 19 15:37:47.549503 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:37:47.549485    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7\": container with ID starting with d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7 not found: ID does not exist" containerID="d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7"
Sep 19 15:37:47.549555 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.549512    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7"} err="failed to get container status \"d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7\": rpc error: code = NotFound desc = could not find container \"d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7\": container with ID starting with d2b689dd549760322412e3d1128fd5b2db376f000f7828e51486f0c1dde742e7 not found: ID does not exist"
Sep 19 15:37:47.565023 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.565000    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"]
Sep 19 15:37:47.570110 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.570091    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/simulator-w3-33737b-kserve-5ffdc5855f-c6fh6"]
Sep 19 15:37:47.581238 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.581218    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"]
Sep 19 15:37:47.585050 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:47.585029    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-33737b/distinct-2-w3-33737b-kserve-8694b4565d-md7tm"]
Sep 19 15:37:48.173907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:48.173876    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32dd25e7-4a03-4ebb-9207-95ebc38b6e84" path="/var/lib/kubelet/pods/32dd25e7-4a03-4ebb-9207-95ebc38b6e84/volumes"
Sep 19 15:37:48.174394 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:48.174378    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" path="/var/lib/kubelet/pods/4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38/volumes"
Sep 19 15:37:50.253799 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:50.253769    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:50.254155 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:50.253811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:50.265787 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:50.265763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:37:50.559700 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:37:50.559628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:38:10.136115 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136085    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"]
Sep 19 15:38:10.136472 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136430    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d0f7d5-2e94-4aaf-be23-85251088c4b4" containerName="payload-processing"
Sep 19 15:38:10.136472 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136442    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d0f7d5-2e94-4aaf-be23-85251088c4b4" containerName="payload-processing"
Sep 19 15:38:10.136472 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136455    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" containerName="main"
Sep 19 15:38:10.136472 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136460    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" containerName="main"
Sep 19 15:38:10.136472 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136470    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32dd25e7-4a03-4ebb-9207-95ebc38b6e84" containerName="main"
Sep 19 15:38:10.136472 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136475    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32dd25e7-4a03-4ebb-9207-95ebc38b6e84" containerName="main"
Sep 19 15:38:10.136684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136484    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f3601e-4fd4-4936-ba9f-f0a25db7d5f6" containerName="payload-pre-processing"
Sep 19 15:38:10.136684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136489    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f3601e-4fd4-4936-ba9f-f0a25db7d5f6" containerName="payload-pre-processing"
Sep 19 15:38:10.136684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136562    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79d0f7d5-2e94-4aaf-be23-85251088c4b4" containerName="payload-processing"
Sep 19 15:38:10.136684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136572    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32dd25e7-4a03-4ebb-9207-95ebc38b6e84" containerName="main"
Sep 19 15:38:10.136684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136580    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f3601e-4fd4-4936-ba9f-f0a25db7d5f6" containerName="payload-pre-processing"
Sep 19 15:38:10.136684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.136587    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fcf6ebd-71ce-4a53-b3c3-0a962bac7f38" containerName="main"
Sep 19 15:38:10.139548 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.139531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.142430 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.142407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9c5c2\"/\"openshift-service-ca.crt\""
Sep 19 15:38:10.142430 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.142418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9c5c2\"/\"auth-test-model-f9c5c2-kserve-self-signed-certs\""
Sep 19 15:38:10.142624 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.142418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9c5c2\"/\"default-dockercfg-zwxth\""
Sep 19 15:38:10.142624 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.142448    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9c5c2\"/\"kube-root-ca.crt\""
Sep 19 15:38:10.147777 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.147757    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"]
Sep 19 15:38:10.187051 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.187024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854ls\" (UniqueName: \"kubernetes.io/projected/59040f93-eceb-406b-b45e-d7d9b40b9b4e-kube-api-access-854ls\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.187154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.187066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-dshm\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.187154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.187086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-home\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.187154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.187103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tls-certs\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.187264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.187163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tmp-dir\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.187264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.187241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-model-cache\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288176 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-home\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288310 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tls-certs\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288310 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tmp-dir\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288310 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-model-cache\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288428 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854ls\" (UniqueName: \"kubernetes.io/projected/59040f93-eceb-406b-b45e-d7d9b40b9b4e-kube-api-access-854ls\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288428 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-dshm\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288630 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tmp-dir\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288710 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-home\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.288758 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.288710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-model-cache\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.290615 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.290587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-dshm\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.290917 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.290897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tls-certs\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.295971 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.295955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-854ls\" (UniqueName: \"kubernetes.io/projected/59040f93-eceb-406b-b45e-d7d9b40b9b4e-kube-api-access-854ls\") pod \"auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.449938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.449905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:10.722797 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:10.722772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"]
Sep 19 15:38:10.725046 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:38:10.725022    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59040f93_eceb_406b_b45e_d7d9b40b9b4e.slice/crio-2907bbce09bd80d07860de6d1202d5be29a1c03564afce3eeee3a2d7c6c84d7c WatchSource:0}: Error finding container 2907bbce09bd80d07860de6d1202d5be29a1c03564afce3eeee3a2d7c6c84d7c: Status 404 returned error can't find the container with id 2907bbce09bd80d07860de6d1202d5be29a1c03564afce3eeee3a2d7c6c84d7c
Sep 19 15:38:11.654087 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:11.654055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw" event={"ID":"59040f93-eceb-406b-b45e-d7d9b40b9b4e","Type":"ContainerStarted","Data":"1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70"}
Sep 19 15:38:11.654439 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:11.654097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw" event={"ID":"59040f93-eceb-406b-b45e-d7d9b40b9b4e","Type":"ContainerStarted","Data":"2907bbce09bd80d07860de6d1202d5be29a1c03564afce3eeee3a2d7c6c84d7c"}
Sep 19 15:38:11.668921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:11.668866    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw" podStartSLOduration=1.668852535 podStartE2EDuration="1.668852535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:38:11.667565686 +0000 UTC m=+1553.561129327" watchObservedRunningTime="2026-09-19 15:38:11.668852535 +0000 UTC m=+1553.562416173"
Sep 19 15:38:20.284240 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:20.284215    2579 scope.go:117] "RemoveContainer" containerID="284001b91877262643b4d7862e327203518f751a7dcc34b369a2bcc31bc31332"
Sep 19 15:38:20.450948 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:20.450919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:20.451043 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:20.450959    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:20.462632 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:20.462611    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:20.710242 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:20.710222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:38:46.610995 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:46.610952    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"]
Sep 19 15:38:46.636189 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:46.636159    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"]
Sep 19 15:38:56.498444 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.498409    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"]
Sep 19 15:38:56.498957 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.498677    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz" podUID="4cb5b58b-de14-47ab-a2cd-9aa963f7e668" containerName="payload-processing" containerID="cri-o://aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90" gracePeriod=30
Sep 19 15:38:56.505468 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.505304    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"]
Sep 19 15:38:56.505594 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.505516    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp" podUID="86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb" containerName="payload-pre-processing" containerID="cri-o://c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd" gracePeriod=30
Sep 19 15:38:56.786806 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.786784    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:38:56.789501 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.789483    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:38:56.849923 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.849890    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-plugins-config-volume\") pod \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\" (UID: \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\") "
Sep 19 15:38:56.850076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.849932    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-plugins-config-volume\") pod \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\" (UID: \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\") "
Sep 19 15:38:56.850076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.849965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvgtg\" (UniqueName: \"kubernetes.io/projected/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-kube-api-access-bvgtg\") pod \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\" (UID: \"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb\") "
Sep 19 15:38:56.850076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.850001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4pzl\" (UniqueName: \"kubernetes.io/projected/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-kube-api-access-q4pzl\") pod \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\" (UID: \"4cb5b58b-de14-47ab-a2cd-9aa963f7e668\") "
Sep 19 15:38:56.850295 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.850268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb" (UID: "86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:38:56.850432 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.850314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cb5b58b-de14-47ab-a2cd-9aa963f7e668" (UID: "4cb5b58b-de14-47ab-a2cd-9aa963f7e668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:38:56.850432 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.850413    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:38:56.852264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.852243    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-kube-api-access-q4pzl" (OuterVolumeSpecName: "kube-api-access-q4pzl") pod "4cb5b58b-de14-47ab-a2cd-9aa963f7e668" (UID: "4cb5b58b-de14-47ab-a2cd-9aa963f7e668"). InnerVolumeSpecName "kube-api-access-q4pzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:38:56.852264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.852258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-kube-api-access-bvgtg" (OuterVolumeSpecName: "kube-api-access-bvgtg") pod "86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb" (UID: "86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb"). InnerVolumeSpecName "kube-api-access-bvgtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:38:56.873038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.873013    2579 generic.go:356] "Generic (PLEG): container finished" podID="86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb" containerID="c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd" exitCode=0
Sep 19 15:38:56.873135 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.873057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"
Sep 19 15:38:56.873135 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.873091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp" event={"ID":"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb","Type":"ContainerDied","Data":"c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd"}
Sep 19 15:38:56.873135 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.873134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp" event={"ID":"86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb","Type":"ContainerDied","Data":"81053f7bed862972f0fa61d067cbae5d0f455f03adbdd2c0cc0dd6fe95cb1899"}
Sep 19 15:38:56.873264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.873158    2579 scope.go:117] "RemoveContainer" containerID="c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd"
Sep 19 15:38:56.874466 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.874445    2579 generic.go:356] "Generic (PLEG): container finished" podID="4cb5b58b-de14-47ab-a2cd-9aa963f7e668" containerID="aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90" exitCode=0
Sep 19 15:38:56.874556 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.874489    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"
Sep 19 15:38:56.874556 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.874524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz" event={"ID":"4cb5b58b-de14-47ab-a2cd-9aa963f7e668","Type":"ContainerDied","Data":"aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90"}
Sep 19 15:38:56.874674 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.874587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz" event={"ID":"4cb5b58b-de14-47ab-a2cd-9aa963f7e668","Type":"ContainerDied","Data":"5e9d93481d6fa99601bae267d05704cea27c303cdbc1fb19fc80548e6d39e390"}
Sep 19 15:38:56.881842 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.881826    2579 scope.go:117] "RemoveContainer" containerID="c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd"
Sep 19 15:38:56.882106 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:38:56.882089    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd\": container with ID starting with c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd not found: ID does not exist" containerID="c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd"
Sep 19 15:38:56.882173 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.882113    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd"} err="failed to get container status \"c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd\": rpc error: code = NotFound desc = could not find container \"c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd\": container with ID starting with c64a877061fdf880f43666f153c981441bb7df30f0f9dc217cb5ec79660ab7bd not found: ID does not exist"
Sep 19 15:38:56.882173 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.882130    2579 scope.go:117] "RemoveContainer" containerID="aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90"
Sep 19 15:38:56.890045 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.890031    2579 scope.go:117] "RemoveContainer" containerID="aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90"
Sep 19 15:38:56.890290 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:38:56.890275    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90\": container with ID starting with aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90 not found: ID does not exist" containerID="aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90"
Sep 19 15:38:56.890328 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.890298    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90"} err="failed to get container status \"aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90\": rpc error: code = NotFound desc = could not find container \"aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90\": container with ID starting with aae80882314bd86fbacf7d61386b653bcd7e6a601a26731c2136e34fea608f90 not found: ID does not exist"
Sep 19 15:38:56.897034 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.897013    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"]
Sep 19 15:38:56.901158 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.901141    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-15949d-5dbdc577fd-9vfgp"]
Sep 19 15:38:56.910297 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.910270    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"]
Sep 19 15:38:56.912076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.912059    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-15949d-5694879547-vbjrz"]
Sep 19 15:38:56.951379 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.951358    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:38:56.951379 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.951377    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvgtg\" (UniqueName: \"kubernetes.io/projected/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb-kube-api-access-bvgtg\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:38:56.951494 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:56.951387    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4pzl\" (UniqueName: \"kubernetes.io/projected/4cb5b58b-de14-47ab-a2cd-9aa963f7e668-kube-api-access-q4pzl\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:38:58.170629 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:58.170584    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb5b58b-de14-47ab-a2cd-9aa963f7e668" path="/var/lib/kubelet/pods/4cb5b58b-de14-47ab-a2cd-9aa963f7e668/volumes"
Sep 19 15:38:58.170967 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:38:58.170953    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb" path="/var/lib/kubelet/pods/86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb/volumes"
Sep 19 15:39:01.630965 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.630882    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds" podUID="c7ea8b18-5c80-43db-8394-a9084e396311" containerName="main" containerID="cri-o://c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367" gracePeriod=45
Sep 19 15:39:01.655274 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.655247    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986" podUID="6e132ed7-f33a-424b-abbf-4d7755336f99" containerName="main" containerID="cri-o://cc22bf6920e65709548afc8795952e4cfb86708662ca4eaa36231181a1cccbdd" gracePeriod=45
Sep 19 15:39:01.897265 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.897238    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:39:01.902562 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.902539    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7ea8b18-5c80-43db-8394-a9084e396311" containerID="c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367" exitCode=0
Sep 19 15:39:01.902712 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.902579    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"
Sep 19 15:39:01.902712 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.902651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds" event={"ID":"c7ea8b18-5c80-43db-8394-a9084e396311","Type":"ContainerDied","Data":"c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367"}
Sep 19 15:39:01.902712 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.902699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds" event={"ID":"c7ea8b18-5c80-43db-8394-a9084e396311","Type":"ContainerDied","Data":"7f3b5a2d11c0b6cfe394533d4c46e56cb3ec95808faa8818ddd11724671a6bba"}
Sep 19 15:39:01.902865 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.902721    2579 scope.go:117] "RemoveContainer" containerID="c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367"
Sep 19 15:39:01.904352 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.904331    2579 generic.go:356] "Generic (PLEG): container finished" podID="6e132ed7-f33a-424b-abbf-4d7755336f99" containerID="cc22bf6920e65709548afc8795952e4cfb86708662ca4eaa36231181a1cccbdd" exitCode=0
Sep 19 15:39:01.904446 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.904365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986" event={"ID":"6e132ed7-f33a-424b-abbf-4d7755336f99","Type":"ContainerDied","Data":"cc22bf6920e65709548afc8795952e4cfb86708662ca4eaa36231181a1cccbdd"}
Sep 19 15:39:01.911749 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.911729    2579 scope.go:117] "RemoveContainer" containerID="c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367"
Sep 19 15:39:01.912096 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:39:01.912075    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367\": container with ID starting with c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367 not found: ID does not exist" containerID="c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367"
Sep 19 15:39:01.912184 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.912107    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367"} err="failed to get container status \"c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367\": rpc error: code = NotFound desc = could not find container \"c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367\": container with ID starting with c6fca2a15d30d16bf3d8ce9dbf3b6a3e8d04f364fb4d73951da2bbff209bf367 not found: ID does not exist"
Sep 19 15:39:01.936046 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.936026    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:39:01.992140 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e132ed7-f33a-424b-abbf-4d7755336f99-tls-certs\") pod \"6e132ed7-f33a-424b-abbf-4d7755336f99\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") "
Sep 19 15:39:01.992296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-tmp-dir\") pod \"6e132ed7-f33a-424b-abbf-4d7755336f99\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") "
Sep 19 15:39:01.992296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ea8b18-5c80-43db-8394-a9084e396311-tls-certs\") pod \"c7ea8b18-5c80-43db-8394-a9084e396311\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") "
Sep 19 15:39:01.992420 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992296    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvbnm\" (UniqueName: \"kubernetes.io/projected/c7ea8b18-5c80-43db-8394-a9084e396311-kube-api-access-xvbnm\") pod \"c7ea8b18-5c80-43db-8394-a9084e396311\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") "
Sep 19 15:39:01.992420 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992327    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-tmp-dir\") pod \"c7ea8b18-5c80-43db-8394-a9084e396311\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") "
Sep 19 15:39:01.992528 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-dshm\") pod \"c7ea8b18-5c80-43db-8394-a9084e396311\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") "
Sep 19 15:39:01.992528 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e132ed7-f33a-424b-abbf-4d7755336f99" (UID: "6e132ed7-f33a-424b-abbf-4d7755336f99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:01.992528 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992466    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-dshm\") pod \"6e132ed7-f33a-424b-abbf-4d7755336f99\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") "
Sep 19 15:39:01.992528 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-home\") pod \"c7ea8b18-5c80-43db-8394-a9084e396311\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") "
Sep 19 15:39:01.992749 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992573    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7qdf\" (UniqueName: \"kubernetes.io/projected/6e132ed7-f33a-424b-abbf-4d7755336f99-kube-api-access-w7qdf\") pod \"6e132ed7-f33a-424b-abbf-4d7755336f99\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") "
Sep 19 15:39:01.992749 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992657    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-model-cache\") pod \"c7ea8b18-5c80-43db-8394-a9084e396311\" (UID: \"c7ea8b18-5c80-43db-8394-a9084e396311\") "
Sep 19 15:39:01.992749 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992698    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-model-cache\") pod \"6e132ed7-f33a-424b-abbf-4d7755336f99\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") "
Sep 19 15:39:01.993076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.992791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-home\") pod \"6e132ed7-f33a-424b-abbf-4d7755336f99\" (UID: \"6e132ed7-f33a-424b-abbf-4d7755336f99\") "
Sep 19 15:39:01.993171 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.993149    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:01.993231 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.993179    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7ea8b18-5c80-43db-8394-a9084e396311" (UID: "c7ea8b18-5c80-43db-8394-a9084e396311"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:01.993682 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.993389    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-home" (OuterVolumeSpecName: "home") pod "6e132ed7-f33a-424b-abbf-4d7755336f99" (UID: "6e132ed7-f33a-424b-abbf-4d7755336f99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:01.993682 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.993430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7ea8b18-5c80-43db-8394-a9084e396311" (UID: "c7ea8b18-5c80-43db-8394-a9084e396311"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:01.993682 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.993618    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e132ed7-f33a-424b-abbf-4d7755336f99" (UID: "6e132ed7-f33a-424b-abbf-4d7755336f99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:01.993682 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.993656    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-home" (OuterVolumeSpecName: "home") pod "c7ea8b18-5c80-43db-8394-a9084e396311" (UID: "c7ea8b18-5c80-43db-8394-a9084e396311"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:01.995443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.995417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ea8b18-5c80-43db-8394-a9084e396311-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7ea8b18-5c80-43db-8394-a9084e396311" (UID: "c7ea8b18-5c80-43db-8394-a9084e396311"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:39:01.995560 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.995443    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e132ed7-f33a-424b-abbf-4d7755336f99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e132ed7-f33a-424b-abbf-4d7755336f99" (UID: "6e132ed7-f33a-424b-abbf-4d7755336f99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:39:01.995644 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.995567    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-dshm" (OuterVolumeSpecName: "dshm") pod "6e132ed7-f33a-424b-abbf-4d7755336f99" (UID: "6e132ed7-f33a-424b-abbf-4d7755336f99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:01.995644 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.995593    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ea8b18-5c80-43db-8394-a9084e396311-kube-api-access-xvbnm" (OuterVolumeSpecName: "kube-api-access-xvbnm") pod "c7ea8b18-5c80-43db-8394-a9084e396311" (UID: "c7ea8b18-5c80-43db-8394-a9084e396311"). InnerVolumeSpecName "kube-api-access-xvbnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:39:01.996054 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.996024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e132ed7-f33a-424b-abbf-4d7755336f99-kube-api-access-w7qdf" (OuterVolumeSpecName: "kube-api-access-w7qdf") pod "6e132ed7-f33a-424b-abbf-4d7755336f99" (UID: "6e132ed7-f33a-424b-abbf-4d7755336f99"). InnerVolumeSpecName "kube-api-access-w7qdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:39:01.996651 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:01.996590    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-dshm" (OuterVolumeSpecName: "dshm") pod "c7ea8b18-5c80-43db-8394-a9084e396311" (UID: "c7ea8b18-5c80-43db-8394-a9084e396311"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:02.093861 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093833    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e132ed7-f33a-424b-abbf-4d7755336f99-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.093861 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093855    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ea8b18-5c80-43db-8394-a9084e396311-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.093861 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093864    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbnm\" (UniqueName: \"kubernetes.io/projected/c7ea8b18-5c80-43db-8394-a9084e396311-kube-api-access-xvbnm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.094038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093874    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.094038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093882    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.094038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093889    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.094038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093895    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.094038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093902    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qdf\" (UniqueName: \"kubernetes.io/projected/6e132ed7-f33a-424b-abbf-4d7755336f99-kube-api-access-w7qdf\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.094038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093910    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ea8b18-5c80-43db-8394-a9084e396311-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.094038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093917    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.094038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.093924    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e132ed7-f33a-424b-abbf-4d7755336f99-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:02.217094 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.217071    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"]
Sep 19 15:39:02.218921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.218901    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/distinct-w0-15949d-kserve-57778947fc-427ds"]
Sep 19 15:39:02.911529 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.911461    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"
Sep 19 15:39:02.911529 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.911490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986" event={"ID":"6e132ed7-f33a-424b-abbf-4d7755336f99","Type":"ContainerDied","Data":"ec332e9053ade703670223e838edd253dfceb6c2b25bcda29fa82df4ee42b184"}
Sep 19 15:39:02.911978 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.911540    2579 scope.go:117] "RemoveContainer" containerID="cc22bf6920e65709548afc8795952e4cfb86708662ca4eaa36231181a1cccbdd"
Sep 19 15:39:02.930701 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.930565    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"]
Sep 19 15:39:02.932500 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:02.932476    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986"]
Sep 19 15:39:04.171995 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:04.171965    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e132ed7-f33a-424b-abbf-4d7755336f99" path="/var/lib/kubelet/pods/6e132ed7-f33a-424b-abbf-4d7755336f99/volumes"
Sep 19 15:39:04.172355 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:04.172322    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7ea8b18-5c80-43db-8394-a9084e396311" path="/var/lib/kubelet/pods/c7ea8b18-5c80-43db-8394-a9084e396311/volumes"
Sep 19 15:39:06.850673 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:06.850637    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-15949d/premium-w0-15949d-kserve-765967dff4-6l986" podUID="6e132ed7-f33a-424b-abbf-4d7755336f99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 19 15:39:08.760593 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:08.760562    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"]
Sep 19 15:39:08.761058 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:08.760791    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2" podUID="49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e" containerName="payload-processing" containerID="cri-o://1e371550f8a427673a6f7406d6f0099966acfb315dbebebc6923bbb46c625d25" gracePeriod=30
Sep 19 15:39:08.945520 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:08.945493    2579 generic.go:356] "Generic (PLEG): container finished" podID="49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e" containerID="1e371550f8a427673a6f7406d6f0099966acfb315dbebebc6923bbb46c625d25" exitCode=0
Sep 19 15:39:08.945704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:08.945565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2" event={"ID":"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e","Type":"ContainerDied","Data":"1e371550f8a427673a6f7406d6f0099966acfb315dbebebc6923bbb46c625d25"}
Sep 19 15:39:09.000970 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.000951    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:39:09.045508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.045448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-plugins-config-volume\") pod \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\" (UID: \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\") "
Sep 19 15:39:09.045638 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.045507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sk2q\" (UniqueName: \"kubernetes.io/projected/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-kube-api-access-6sk2q\") pod \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\" (UID: \"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e\") "
Sep 19 15:39:09.045794 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.045771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e" (UID: "49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:39:09.047575 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.047556    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-kube-api-access-6sk2q" (OuterVolumeSpecName: "kube-api-access-6sk2q") pod "49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e" (UID: "49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e"). InnerVolumeSpecName "kube-api-access-6sk2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:39:09.146834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.146810    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:09.146834 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.146834    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sk2q\" (UniqueName: \"kubernetes.io/projected/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e-kube-api-access-6sk2q\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:09.952476 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.952446    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"
Sep 19 15:39:09.952951 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.952474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2" event={"ID":"49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e","Type":"ContainerDied","Data":"cb29508d537a6f0f5e1dcff06abe417045a15f7c95cf3367fb390e8488fba2ea"}
Sep 19 15:39:09.952951 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.952568    2579 scope.go:117] "RemoveContainer" containerID="1e371550f8a427673a6f7406d6f0099966acfb315dbebebc6923bbb46c625d25"
Sep 19 15:39:09.975802 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.975780    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"]
Sep 19 15:39:09.977911 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:09.977887    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7cbcece2-69b868bd6f-48vl2"]
Sep 19 15:39:10.170360 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:10.170334    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e" path="/var/lib/kubelet/pods/49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e/volumes"
Sep 19 15:39:16.942261 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:16.942227    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"]
Sep 19 15:39:20.316789 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:20.316765    2579 scope.go:117] "RemoveContainer" containerID="be772ea8356ebd53fc9447687aabe8db535407ba5b0af60e3cb9ec26152d93b2"
Sep 19 15:39:31.961210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:31.961161    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl" podUID="3ea9189a-18dc-443d-ad45-1030bc5f5271" containerName="main" containerID="cri-o://2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46" gracePeriod=45
Sep 19 15:39:32.213484 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.213436    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:39:32.327398 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjqs6\" (UniqueName: \"kubernetes.io/projected/3ea9189a-18dc-443d-ad45-1030bc5f5271-kube-api-access-gjqs6\") pod \"3ea9189a-18dc-443d-ad45-1030bc5f5271\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") "
Sep 19 15:39:32.327551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea9189a-18dc-443d-ad45-1030bc5f5271-tls-certs\") pod \"3ea9189a-18dc-443d-ad45-1030bc5f5271\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") "
Sep 19 15:39:32.327551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327445    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-dshm\") pod \"3ea9189a-18dc-443d-ad45-1030bc5f5271\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") "
Sep 19 15:39:32.327551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-tmp-dir\") pod \"3ea9189a-18dc-443d-ad45-1030bc5f5271\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") "
Sep 19 15:39:32.327551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327540    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-model-cache\") pod \"3ea9189a-18dc-443d-ad45-1030bc5f5271\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") "
Sep 19 15:39:32.327802 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327581    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-home\") pod \"3ea9189a-18dc-443d-ad45-1030bc5f5271\" (UID: \"3ea9189a-18dc-443d-ad45-1030bc5f5271\") "
Sep 19 15:39:32.327870 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ea9189a-18dc-443d-ad45-1030bc5f5271" (UID: "3ea9189a-18dc-443d-ad45-1030bc5f5271"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:32.327922 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ea9189a-18dc-443d-ad45-1030bc5f5271" (UID: "3ea9189a-18dc-443d-ad45-1030bc5f5271"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:32.327975 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-home" (OuterVolumeSpecName: "home") pod "3ea9189a-18dc-443d-ad45-1030bc5f5271" (UID: "3ea9189a-18dc-443d-ad45-1030bc5f5271"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:32.327975 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.327954    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:32.329725 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.329702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-dshm" (OuterVolumeSpecName: "dshm") pod "3ea9189a-18dc-443d-ad45-1030bc5f5271" (UID: "3ea9189a-18dc-443d-ad45-1030bc5f5271"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:39:32.329816 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.329790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea9189a-18dc-443d-ad45-1030bc5f5271-kube-api-access-gjqs6" (OuterVolumeSpecName: "kube-api-access-gjqs6") pod "3ea9189a-18dc-443d-ad45-1030bc5f5271" (UID: "3ea9189a-18dc-443d-ad45-1030bc5f5271"). InnerVolumeSpecName "kube-api-access-gjqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:39:32.329895 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.329874    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea9189a-18dc-443d-ad45-1030bc5f5271-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ea9189a-18dc-443d-ad45-1030bc5f5271" (UID: "3ea9189a-18dc-443d-ad45-1030bc5f5271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:39:32.428487 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.428467    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea9189a-18dc-443d-ad45-1030bc5f5271-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:32.428487 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.428485    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:32.428622 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.428493    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:32.428622 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.428502    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea9189a-18dc-443d-ad45-1030bc5f5271-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:32.428622 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:32.428510    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqs6\" (UniqueName: \"kubernetes.io/projected/3ea9189a-18dc-443d-ad45-1030bc5f5271-kube-api-access-gjqs6\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:39:33.066571 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.066545    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ea9189a-18dc-443d-ad45-1030bc5f5271" containerID="2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46" exitCode=0
Sep 19 15:39:33.066953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.066593    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"
Sep 19 15:39:33.066953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.066594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl" event={"ID":"3ea9189a-18dc-443d-ad45-1030bc5f5271","Type":"ContainerDied","Data":"2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46"}
Sep 19 15:39:33.066953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.066645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl" event={"ID":"3ea9189a-18dc-443d-ad45-1030bc5f5271","Type":"ContainerDied","Data":"b9d15f3f98bc9e0aed9aef55c3c82a7f0169de19b2a2db11a4a284eaa7ff8d3b"}
Sep 19 15:39:33.066953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.066662    2579 scope.go:117] "RemoveContainer" containerID="2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46"
Sep 19 15:39:33.075394 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.075375    2579 scope.go:117] "RemoveContainer" containerID="2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46"
Sep 19 15:39:33.075700 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:39:33.075679    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46\": container with ID starting with 2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46 not found: ID does not exist" containerID="2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46"
Sep 19 15:39:33.075781 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.075706    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46"} err="failed to get container status \"2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46\": rpc error: code = NotFound desc = could not find container \"2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46\": container with ID starting with 2559be0d594d99824a7f0f13c4d1ad96e88e443b2d025d57cef81db6038a4e46 not found: ID does not exist"
Sep 19 15:39:33.085099 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.085080    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"]
Sep 19 15:39:33.088335 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:33.088317    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7cbcece2/e2e-lifecycle-model-7cbcece2-kserve-6f864dfc7f-jwfvl"]
Sep 19 15:39:34.170749 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:39:34.170721    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea9189a-18dc-443d-ad45-1030bc5f5271" path="/var/lib/kubelet/pods/3ea9189a-18dc-443d-ad45-1030bc5f5271/volumes"
Sep 19 15:40:46.733174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:40:46.733103    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"]
Sep 19 15:41:01.752327 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:01.752268    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw" podUID="59040f93-eceb-406b-b45e-d7d9b40b9b4e" containerName="main" containerID="cri-o://1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70" gracePeriod=45
Sep 19 15:41:02.002575 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.002527    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:41:02.113267 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113244    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-model-cache\") pod \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") "
Sep 19 15:41:02.113387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113306    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tmp-dir\") pod \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") "
Sep 19 15:41:02.113387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tls-certs\") pod \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") "
Sep 19 15:41:02.113504 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-dshm\") pod \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") "
Sep 19 15:41:02.113504 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113436    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-home\") pod \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") "
Sep 19 15:41:02.113504 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113463    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-854ls\" (UniqueName: \"kubernetes.io/projected/59040f93-eceb-406b-b45e-d7d9b40b9b4e-kube-api-access-854ls\") pod \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\" (UID: \"59040f93-eceb-406b-b45e-d7d9b40b9b4e\") "
Sep 19 15:41:02.113504 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "59040f93-eceb-406b-b45e-d7d9b40b9b4e" (UID: "59040f93-eceb-406b-b45e-d7d9b40b9b4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:41:02.113739 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59040f93-eceb-406b-b45e-d7d9b40b9b4e" (UID: "59040f93-eceb-406b-b45e-d7d9b40b9b4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:41:02.113739 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113736    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:41:02.113852 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-home" (OuterVolumeSpecName: "home") pod "59040f93-eceb-406b-b45e-d7d9b40b9b4e" (UID: "59040f93-eceb-406b-b45e-d7d9b40b9b4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:41:02.113852 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.113748    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:41:02.115759 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.115733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-dshm" (OuterVolumeSpecName: "dshm") pod "59040f93-eceb-406b-b45e-d7d9b40b9b4e" (UID: "59040f93-eceb-406b-b45e-d7d9b40b9b4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:41:02.115858 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.115812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59040f93-eceb-406b-b45e-d7d9b40b9b4e-kube-api-access-854ls" (OuterVolumeSpecName: "kube-api-access-854ls") pod "59040f93-eceb-406b-b45e-d7d9b40b9b4e" (UID: "59040f93-eceb-406b-b45e-d7d9b40b9b4e"). InnerVolumeSpecName "kube-api-access-854ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:41:02.115858 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.115825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59040f93-eceb-406b-b45e-d7d9b40b9b4e" (UID: "59040f93-eceb-406b-b45e-d7d9b40b9b4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:41:02.214288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.214263    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040f93-eceb-406b-b45e-d7d9b40b9b4e-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:41:02.214288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.214284    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:41:02.214288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.214292    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040f93-eceb-406b-b45e-d7d9b40b9b4e-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:41:02.214445 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.214300    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854ls\" (UniqueName: \"kubernetes.io/projected/59040f93-eceb-406b-b45e-d7d9b40b9b4e-kube-api-access-854ls\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:41:02.471393 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.471366    2579 generic.go:356] "Generic (PLEG): container finished" podID="59040f93-eceb-406b-b45e-d7d9b40b9b4e" containerID="1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70" exitCode=0
Sep 19 15:41:02.471506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.471406    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"
Sep 19 15:41:02.471506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.471449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw" event={"ID":"59040f93-eceb-406b-b45e-d7d9b40b9b4e","Type":"ContainerDied","Data":"1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70"}
Sep 19 15:41:02.471506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.471483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw" event={"ID":"59040f93-eceb-406b-b45e-d7d9b40b9b4e","Type":"ContainerDied","Data":"2907bbce09bd80d07860de6d1202d5be29a1c03564afce3eeee3a2d7c6c84d7c"}
Sep 19 15:41:02.471506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.471500    2579 scope.go:117] "RemoveContainer" containerID="1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70"
Sep 19 15:41:02.479657 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.479641    2579 scope.go:117] "RemoveContainer" containerID="1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70"
Sep 19 15:41:02.479908 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:41:02.479891    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70\": container with ID starting with 1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70 not found: ID does not exist" containerID="1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70"
Sep 19 15:41:02.479954 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.479916    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70"} err="failed to get container status \"1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70\": rpc error: code = NotFound desc = could not find container \"1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70\": container with ID starting with 1189836675ad2ddb3a4e1bf63cad7f74161faec91dea411efa6abb14ced5ce70 not found: ID does not exist"
Sep 19 15:41:02.485914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.485886    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"]
Sep 19 15:41:02.487650 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:02.487629    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/auth-test-model-f9c5c2-kserve-7d664f74b7-wnxrw"]
Sep 19 15:41:04.170532 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:04.170498    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59040f93-eceb-406b-b45e-d7d9b40b9b4e" path="/var/lib/kubelet/pods/59040f93-eceb-406b-b45e-d7d9b40b9b4e/volumes"
Sep 19 15:41:13.476188 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476158    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"]
Sep 19 15:41:13.476545 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476521    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e132ed7-f33a-424b-abbf-4d7755336f99" containerName="main"
Sep 19 15:41:13.476545 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476533    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e132ed7-f33a-424b-abbf-4d7755336f99" containerName="main"
Sep 19 15:41:13.476545 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476544    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb5b58b-de14-47ab-a2cd-9aa963f7e668" containerName="payload-processing"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476550    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb5b58b-de14-47ab-a2cd-9aa963f7e668" containerName="payload-processing"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476558    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb" containerName="payload-pre-processing"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476563    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb" containerName="payload-pre-processing"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476582    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e" containerName="payload-processing"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476590    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e" containerName="payload-processing"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476614    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ea8b18-5c80-43db-8394-a9084e396311" containerName="main"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476622    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ea8b18-5c80-43db-8394-a9084e396311" containerName="main"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476631    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea9189a-18dc-443d-ad45-1030bc5f5271" containerName="main"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476638    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea9189a-18dc-443d-ad45-1030bc5f5271" containerName="main"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476657    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59040f93-eceb-406b-b45e-d7d9b40b9b4e" containerName="main"
Sep 19 15:41:13.476671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476666    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="59040f93-eceb-406b-b45e-d7d9b40b9b4e" containerName="main"
Sep 19 15:41:13.476982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476726    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e6ddcf-2f9a-44ea-b4cc-c7ccc9b86b5e" containerName="payload-processing"
Sep 19 15:41:13.476982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476737    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="59040f93-eceb-406b-b45e-d7d9b40b9b4e" containerName="main"
Sep 19 15:41:13.476982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476746    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb5b58b-de14-47ab-a2cd-9aa963f7e668" containerName="payload-processing"
Sep 19 15:41:13.476982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476754    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ae52b9-f2a5-404f-a42d-25e2b0cd4ddb" containerName="payload-pre-processing"
Sep 19 15:41:13.476982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476760    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e132ed7-f33a-424b-abbf-4d7755336f99" containerName="main"
Sep 19 15:41:13.476982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476765    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7ea8b18-5c80-43db-8394-a9084e396311" containerName="main"
Sep 19 15:41:13.476982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.476771    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea9189a-18dc-443d-ad45-1030bc5f5271" containerName="main"
Sep 19 15:41:13.479858 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.479840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.481737 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.481712    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-88b9ff-serving-cert\""
Sep 19 15:41:13.482443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.482429    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 15:41:13.482553 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.482537    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zhfqt\""
Sep 19 15:41:13.482638 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.482542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 15:41:13.482638 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.482554    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 15:41:13.486585 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.486559    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"]
Sep 19 15:41:13.602647 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.602593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-openshift-service-ca\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.602647 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.602645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-maas-api-tls\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.602812 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.602698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjqqq\" (UniqueName: \"kubernetes.io/projected/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-kube-api-access-zjqqq\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.602812 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.602739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-metrics-certs\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.703693 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.703664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-openshift-service-ca\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.703808 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.703696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-maas-api-tls\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.703808 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.703735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjqqq\" (UniqueName: \"kubernetes.io/projected/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-kube-api-access-zjqqq\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.703808 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.703772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-metrics-certs\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.704314 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.704291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-openshift-service-ca\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.706174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.706155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-maas-api-tls\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.706264 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.706237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-metrics-certs\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.710519 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.710499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjqqq\" (UniqueName: \"kubernetes.io/projected/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-kube-api-access-zjqqq\") pod \"maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:13.791213 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:13.791154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:14.059874 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.059847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"]
Sep 19 15:41:14.062531 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:41:14.062500    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bed351e_cf1d_4900_b4e1_9d8e46f9d80a.slice/crio-7b9fe7c3d2128eca7fc504de992a0c7a0980f75abbb0b2e519ea72471eddc4d5 WatchSource:0}: Error finding container 7b9fe7c3d2128eca7fc504de992a0c7a0980f75abbb0b2e519ea72471eddc4d5: Status 404 returned error can't find the container with id 7b9fe7c3d2128eca7fc504de992a0c7a0980f75abbb0b2e519ea72471eddc4d5
Sep 19 15:41:14.064038 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.064015    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 15:41:14.171571 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.171544    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"]
Sep 19 15:41:14.176022 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.176005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:14.177867 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.177843    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-88b9ff\""
Sep 19 15:41:14.178195 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.178180    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-88b9ff-dockercfg-5x6sp\""
Sep 19 15:41:14.182685 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.182666    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"]
Sep 19 15:41:14.206020 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.206000    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"]
Sep 19 15:41:14.209291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.209276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:14.219434 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.219411    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"]
Sep 19 15:41:14.309626 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.309545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxxxv\" (UniqueName: \"kubernetes.io/projected/6e16b533-e89f-4c16-bd69-98e92d2ca43d-kube-api-access-gxxxv\") pod \"payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59\" (UID: \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\") " pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:14.309626 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.309575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a47d53-96bf-459d-a3f6-0f8512c8732a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84\" (UID: \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:14.309784 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.309653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glmxz\" (UniqueName: \"kubernetes.io/projected/e6a47d53-96bf-459d-a3f6-0f8512c8732a-kube-api-access-glmxz\") pod \"payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84\" (UID: \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:14.309784 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.309687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16b533-e89f-4c16-bd69-98e92d2ca43d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59\" (UID: \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\") " pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:14.410847 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.410820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxxxv\" (UniqueName: \"kubernetes.io/projected/6e16b533-e89f-4c16-bd69-98e92d2ca43d-kube-api-access-gxxxv\") pod \"payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59\" (UID: \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\") " pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:14.410953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.410852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a47d53-96bf-459d-a3f6-0f8512c8732a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84\" (UID: \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:14.410953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.410891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glmxz\" (UniqueName: \"kubernetes.io/projected/e6a47d53-96bf-459d-a3f6-0f8512c8732a-kube-api-access-glmxz\") pod \"payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84\" (UID: \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:14.410953 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.410915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16b533-e89f-4c16-bd69-98e92d2ca43d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59\" (UID: \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\") " pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:14.411579 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.411561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a47d53-96bf-459d-a3f6-0f8512c8732a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84\" (UID: \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:14.411648 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.411562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16b533-e89f-4c16-bd69-98e92d2ca43d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59\" (UID: \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\") " pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:14.418143 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.418115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxxxv\" (UniqueName: \"kubernetes.io/projected/6e16b533-e89f-4c16-bd69-98e92d2ca43d-kube-api-access-gxxxv\") pod \"payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59\" (UID: \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\") " pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:14.418299 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.418281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glmxz\" (UniqueName: \"kubernetes.io/projected/e6a47d53-96bf-459d-a3f6-0f8512c8732a-kube-api-access-glmxz\") pod \"payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84\" (UID: \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:14.487165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.487148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:14.518429 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.518406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:14.536697 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.536638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5" event={"ID":"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a","Type":"ContainerStarted","Data":"7b9fe7c3d2128eca7fc504de992a0c7a0980f75abbb0b2e519ea72471eddc4d5"}
Sep 19 15:41:14.759344 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.759314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"]
Sep 19 15:41:14.762216 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:41:14.762180    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a47d53_96bf_459d_a3f6_0f8512c8732a.slice/crio-7ec3febb8e93405cabade27295460c34911e68deca54591724ba08dad3010da2 WatchSource:0}: Error finding container 7ec3febb8e93405cabade27295460c34911e68deca54591724ba08dad3010da2: Status 404 returned error can't find the container with id 7ec3febb8e93405cabade27295460c34911e68deca54591724ba08dad3010da2
Sep 19 15:41:14.802402 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:14.802376    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"]
Sep 19 15:41:14.805044 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:41:14.804771    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e16b533_e89f_4c16_bd69_98e92d2ca43d.slice/crio-356c1f93274e792ddafd16784d7e677085c462ac431d6f00f600bbf481de4d46 WatchSource:0}: Error finding container 356c1f93274e792ddafd16784d7e677085c462ac431d6f00f600bbf481de4d46: Status 404 returned error can't find the container with id 356c1f93274e792ddafd16784d7e677085c462ac431d6f00f600bbf481de4d46
Sep 19 15:41:15.544704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:15.544666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59" event={"ID":"6e16b533-e89f-4c16-bd69-98e92d2ca43d","Type":"ContainerStarted","Data":"6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1"}
Sep 19 15:41:15.545168 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:15.544732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59" event={"ID":"6e16b533-e89f-4c16-bd69-98e92d2ca43d","Type":"ContainerStarted","Data":"356c1f93274e792ddafd16784d7e677085c462ac431d6f00f600bbf481de4d46"}
Sep 19 15:41:15.545168 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:15.544763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:15.546971 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:15.546863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84" event={"ID":"e6a47d53-96bf-459d-a3f6-0f8512c8732a","Type":"ContainerStarted","Data":"a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7"}
Sep 19 15:41:15.546971 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:15.546919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84" event={"ID":"e6a47d53-96bf-459d-a3f6-0f8512c8732a","Type":"ContainerStarted","Data":"7ec3febb8e93405cabade27295460c34911e68deca54591724ba08dad3010da2"}
Sep 19 15:41:15.547480 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:15.547431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:15.561779 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:15.561740    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59" podStartSLOduration=1.5617237290000001 podStartE2EDuration="1.561723729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:41:15.558750837 +0000 UTC m=+1737.452314484" watchObservedRunningTime="2026-09-19 15:41:15.561723729 +0000 UTC m=+1737.455287372"
Sep 19 15:41:15.573486 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:15.573434    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84" podStartSLOduration=1.573418499 podStartE2EDuration="1.573418499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:41:15.571823157 +0000 UTC m=+1737.465386787" watchObservedRunningTime="2026-09-19 15:41:15.573418499 +0000 UTC m=+1737.466982138"
Sep 19 15:41:16.554986 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:16.554954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5" event={"ID":"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a","Type":"ContainerStarted","Data":"73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6"}
Sep 19 15:41:16.571447 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:16.571403    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5" podStartSLOduration=1.181289189 podStartE2EDuration="3.571387235s" totalImagesPullingTime="2.390098046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:41:16.569083749 +0000 UTC m=+1738.462647427" watchObservedRunningTime="2026-09-19 15:41:16.571387235 +0000 UTC m=+1738.464950874"
Sep 19 15:41:17.564506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:17.564472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:23.574021 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:23.573989    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:41:26.557081 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:26.557053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:41:27.566329 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:27.566300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:41:48.833541 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.833512    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"]
Sep 19 15:41:48.840576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.840550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.842721 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.842463    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-26cbe6-serving-cert\""
Sep 19 15:41:48.844634 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.844583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"]
Sep 19 15:41:48.878426 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.878407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-openshift-service-ca\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.878522 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.878436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-metrics-certs\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.878522 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.878475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.878522 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.878499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbfkk\" (UniqueName: \"kubernetes.io/projected/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-kube-api-access-hbfkk\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.979562 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.979540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.979675 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.979567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfkk\" (UniqueName: \"kubernetes.io/projected/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-kube-api-access-hbfkk\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.979675 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.979655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-openshift-service-ca\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.979770 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.979675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-metrics-certs\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.979770 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:41:48.979685    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-serving-cert: secret "maas-api-e2e-api-b-26cbe6-serving-cert" not found
Sep 19 15:41:48.979770 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:41:48.979764    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls podName:1bd1c978-4b0d-4e92-95ab-4f81bdd5da87 nodeName:}" failed. No retries permitted until 2026-09-19 15:41:49.479734093 +0000 UTC m=+1771.373297725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls") pod "maas-api-e2e-api-b-26cbe6-657854c945-826rj" (UID: "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87") : secret "maas-api-e2e-api-b-26cbe6-serving-cert" not found
Sep 19 15:41:48.980443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.980418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-openshift-service-ca\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.982078 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.982054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-metrics-certs\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:48.987595 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:48.987576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbfkk\" (UniqueName: \"kubernetes.io/projected/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-kube-api-access-hbfkk\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:49.483130 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.483099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:49.485635 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.485592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls\") pod \"maas-api-e2e-api-b-26cbe6-657854c945-826rj\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:49.537078 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.537046    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"]
Sep 19 15:41:49.540864 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.540848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:49.542890 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.542856    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-26cbe6-dockercfg-qgmsz\""
Sep 19 15:41:49.542890 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.542861    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-26cbe6\""
Sep 19 15:41:49.548646 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.548621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"]
Sep 19 15:41:49.584083 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.584052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58k9j\" (UniqueName: \"kubernetes.io/projected/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-kube-api-access-58k9j\") pod \"payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb\" (UID: \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\") " pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:49.584214 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.584193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb\" (UID: \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\") " pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:49.685003 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.684975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb\" (UID: \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\") " pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:49.685118 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.685067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58k9j\" (UniqueName: \"kubernetes.io/projected/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-kube-api-access-58k9j\") pod \"payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb\" (UID: \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\") " pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:49.685661 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.685639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb\" (UID: \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\") " pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:49.692323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.692297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58k9j\" (UniqueName: \"kubernetes.io/projected/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-kube-api-access-58k9j\") pod \"payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb\" (UID: \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\") " pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:49.753237 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.753184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:49.852977 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:49.852947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:50.020717 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.020692    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"]
Sep 19 15:41:50.022676 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:41:50.022651    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd1c978_4b0d_4e92_95ab_4f81bdd5da87.slice/crio-0f687c7be4ef3da88217f3ca2aa541af278fc67bcc598a6642c8a9247d6487b8 WatchSource:0}: Error finding container 0f687c7be4ef3da88217f3ca2aa541af278fc67bcc598a6642c8a9247d6487b8: Status 404 returned error can't find the container with id 0f687c7be4ef3da88217f3ca2aa541af278fc67bcc598a6642c8a9247d6487b8
Sep 19 15:41:50.119814 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.119794    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"]
Sep 19 15:41:50.121254 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:41:50.121230    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf48da61_061b_4c6f_90b9_e05e1d3d2a0d.slice/crio-08b91079875f88767a7b8274da388c256014db1280cd3c899d5cfaf71b91c823 WatchSource:0}: Error finding container 08b91079875f88767a7b8274da388c256014db1280cd3c899d5cfaf71b91c823: Status 404 returned error can't find the container with id 08b91079875f88767a7b8274da388c256014db1280cd3c899d5cfaf71b91c823
Sep 19 15:41:50.722036 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.722003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb" event={"ID":"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d","Type":"ContainerStarted","Data":"674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f"}
Sep 19 15:41:50.722225 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.722108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:41:50.722225 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.722154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb" event={"ID":"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d","Type":"ContainerStarted","Data":"08b91079875f88767a7b8274da388c256014db1280cd3c899d5cfaf71b91c823"}
Sep 19 15:41:50.723650 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.723627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj" event={"ID":"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87","Type":"ContainerStarted","Data":"2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0"}
Sep 19 15:41:50.723784 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.723667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj" event={"ID":"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87","Type":"ContainerStarted","Data":"0f687c7be4ef3da88217f3ca2aa541af278fc67bcc598a6642c8a9247d6487b8"}
Sep 19 15:41:50.723784 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.723749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:41:50.738443 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.738396    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb" podStartSLOduration=1.738380195 podStartE2EDuration="1.738380195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:41:50.735252043 +0000 UTC m=+1772.628815684" watchObservedRunningTime="2026-09-19 15:41:50.738380195 +0000 UTC m=+1772.631943841"
Sep 19 15:41:50.750323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:50.750275    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj" podStartSLOduration=2.460316006 podStartE2EDuration="2.750259435s" totalImagesPullingTime="289.943429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:41:50.749236529 +0000 UTC m=+1772.642800174" watchObservedRunningTime="2026-09-19 15:41:50.750259435 +0000 UTC m=+1772.643823092"
Sep 19 15:41:56.734228 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:41:56.734198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:42:01.731691 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:01.731587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:42:20.224509 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:20.224481    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:42:20.228861 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:20.228840    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:42:20.371835 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:20.371811    2579 scope.go:117] "RemoveContainer" containerID="3b02a298cd20ea29c09e23362393846cc5d21bca41797d2ada7cf3dfde829652"
Sep 19 15:42:33.527319 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.527284    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"]
Sep 19 15:42:33.527870 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.527481    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5" podUID="8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" containerName="maas-api" containerID="cri-o://73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6" gracePeriod=30
Sep 19 15:42:33.570051 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.570018    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5" podUID="8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.66:8443/health\": dial tcp 10.134.0.66:8443: connect: connection refused"
Sep 19 15:42:33.583170 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.583141    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"]
Sep 19 15:42:33.583393 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.583355    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59" podUID="6e16b533-e89f-4c16-bd69-98e92d2ca43d" containerName="payload-processing" containerID="cri-o://6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1" gracePeriod=30
Sep 19 15:42:33.588263 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.588239    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"]
Sep 19 15:42:33.588567 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.588498    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84" podUID="e6a47d53-96bf-459d-a3f6-0f8512c8732a" containerName="payload-pre-processing" containerID="cri-o://a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7" gracePeriod=30
Sep 19 15:42:33.843252 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.843229    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:42:33.847709 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.847686    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:42:33.867340 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.867310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:42:33.939239 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939201    2579 generic.go:356] "Generic (PLEG): container finished" podID="6e16b533-e89f-4c16-bd69-98e92d2ca43d" containerID="6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1" exitCode=0
Sep 19 15:42:33.939403 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939256    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"
Sep 19 15:42:33.939403 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59" event={"ID":"6e16b533-e89f-4c16-bd69-98e92d2ca43d","Type":"ContainerDied","Data":"6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1"}
Sep 19 15:42:33.939403 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59" event={"ID":"6e16b533-e89f-4c16-bd69-98e92d2ca43d","Type":"ContainerDied","Data":"356c1f93274e792ddafd16784d7e677085c462ac431d6f00f600bbf481de4d46"}
Sep 19 15:42:33.939403 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939363    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16b533-e89f-4c16-bd69-98e92d2ca43d-plugins-config-volume\") pod \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\" (UID: \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\") "
Sep 19 15:42:33.939403 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939386    2579 scope.go:117] "RemoveContainer" containerID="6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1"
Sep 19 15:42:33.939662 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939446    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-openshift-service-ca\") pod \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") "
Sep 19 15:42:33.939662 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-metrics-certs\") pod \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") "
Sep 19 15:42:33.939662 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-maas-api-tls\") pod \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") "
Sep 19 15:42:33.939662 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939564    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjqqq\" (UniqueName: \"kubernetes.io/projected/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-kube-api-access-zjqqq\") pod \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\" (UID: \"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a\") "
Sep 19 15:42:33.939662 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxxxv\" (UniqueName: \"kubernetes.io/projected/6e16b533-e89f-4c16-bd69-98e92d2ca43d-kube-api-access-gxxxv\") pod \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\" (UID: \"6e16b533-e89f-4c16-bd69-98e92d2ca43d\") "
Sep 19 15:42:33.939662 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a47d53-96bf-459d-a3f6-0f8512c8732a-plugins-config-volume\") pod \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\" (UID: \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\") "
Sep 19 15:42:33.939913 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939712    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glmxz\" (UniqueName: \"kubernetes.io/projected/e6a47d53-96bf-459d-a3f6-0f8512c8732a-kube-api-access-glmxz\") pod \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\" (UID: \"e6a47d53-96bf-459d-a3f6-0f8512c8732a\") "
Sep 19 15:42:33.939913 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.939837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e16b533-e89f-4c16-bd69-98e92d2ca43d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e16b533-e89f-4c16-bd69-98e92d2ca43d" (UID: "6e16b533-e89f-4c16-bd69-98e92d2ca43d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:42:33.940040 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.940020    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e16b533-e89f-4c16-bd69-98e92d2ca43d-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:33.940103 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.940061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" (UID: "8bed351e-cf1d-4900-b4e1-9d8e46f9d80a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:42:33.940543 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.940514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a47d53-96bf-459d-a3f6-0f8512c8732a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6a47d53-96bf-459d-a3f6-0f8512c8732a" (UID: "e6a47d53-96bf-459d-a3f6-0f8512c8732a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:42:33.941368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.941144    2579 generic.go:356] "Generic (PLEG): container finished" podID="e6a47d53-96bf-459d-a3f6-0f8512c8732a" containerID="a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7" exitCode=0
Sep 19 15:42:33.941368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.941184    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"
Sep 19 15:42:33.941368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.941293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84" event={"ID":"e6a47d53-96bf-459d-a3f6-0f8512c8732a","Type":"ContainerDied","Data":"a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7"}
Sep 19 15:42:33.941368 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.941329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84" event={"ID":"e6a47d53-96bf-459d-a3f6-0f8512c8732a","Type":"ContainerDied","Data":"7ec3febb8e93405cabade27295460c34911e68deca54591724ba08dad3010da2"}
Sep 19 15:42:33.942769 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.942732    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a47d53-96bf-459d-a3f6-0f8512c8732a-kube-api-access-glmxz" (OuterVolumeSpecName: "kube-api-access-glmxz") pod "e6a47d53-96bf-459d-a3f6-0f8512c8732a" (UID: "e6a47d53-96bf-459d-a3f6-0f8512c8732a"). InnerVolumeSpecName "kube-api-access-glmxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:42:33.942935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.942845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e16b533-e89f-4c16-bd69-98e92d2ca43d-kube-api-access-gxxxv" (OuterVolumeSpecName: "kube-api-access-gxxxv") pod "6e16b533-e89f-4c16-bd69-98e92d2ca43d" (UID: "6e16b533-e89f-4c16-bd69-98e92d2ca43d"). InnerVolumeSpecName "kube-api-access-gxxxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:42:33.942935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.942917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" (UID: "8bed351e-cf1d-4900-b4e1-9d8e46f9d80a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:42:33.943053 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.942956    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" containerID="73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6" exitCode=0
Sep 19 15:42:33.943053 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.942984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5" event={"ID":"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a","Type":"ContainerDied","Data":"73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6"}
Sep 19 15:42:33.943053 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.942996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" (UID: "8bed351e-cf1d-4900-b4e1-9d8e46f9d80a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:42:33.943053 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.942997    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"
Sep 19 15:42:33.943053 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.943018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5" event={"ID":"8bed351e-cf1d-4900-b4e1-9d8e46f9d80a","Type":"ContainerDied","Data":"7b9fe7c3d2128eca7fc504de992a0c7a0980f75abbb0b2e519ea72471eddc4d5"}
Sep 19 15:42:33.943291 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.943273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-kube-api-access-zjqqq" (OuterVolumeSpecName: "kube-api-access-zjqqq") pod "8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" (UID: "8bed351e-cf1d-4900-b4e1-9d8e46f9d80a"). InnerVolumeSpecName "kube-api-access-zjqqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:42:33.949005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.948988    2579 scope.go:117] "RemoveContainer" containerID="6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1"
Sep 19 15:42:33.949267 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:42:33.949250    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1\": container with ID starting with 6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1 not found: ID does not exist" containerID="6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1"
Sep 19 15:42:33.949320 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.949275    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1"} err="failed to get container status \"6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1\": rpc error: code = NotFound desc = could not find container \"6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1\": container with ID starting with 6243058192eb784ddc4027afd5a8dae99976ffcf8fd68c05eba269395d457fb1 not found: ID does not exist"
Sep 19 15:42:33.949320 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.949292    2579 scope.go:117] "RemoveContainer" containerID="a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7"
Sep 19 15:42:33.956491 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.956477    2579 scope.go:117] "RemoveContainer" containerID="a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7"
Sep 19 15:42:33.956732 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:42:33.956716    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7\": container with ID starting with a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7 not found: ID does not exist" containerID="a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7"
Sep 19 15:42:33.956783 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.956739    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7"} err="failed to get container status \"a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7\": rpc error: code = NotFound desc = could not find container \"a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7\": container with ID starting with a0537b4b8a5b318f68bfc2a5e13698b84dfcb03d43eb65ad8c4544fbb535d9f7 not found: ID does not exist"
Sep 19 15:42:33.956783 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.956752    2579 scope.go:117] "RemoveContainer" containerID="73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6"
Sep 19 15:42:33.964360 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.964344    2579 scope.go:117] "RemoveContainer" containerID="73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6"
Sep 19 15:42:33.964626 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:42:33.964589    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6\": container with ID starting with 73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6 not found: ID does not exist" containerID="73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6"
Sep 19 15:42:33.964677 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:33.964632    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6"} err="failed to get container status \"73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6\": rpc error: code = NotFound desc = could not find container \"73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6\": container with ID starting with 73ef1bb2d93596132fa9bc7d5add0fd3d120dc47874a6f5b880d173f5539ebe6 not found: ID does not exist"
Sep 19 15:42:34.041178 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.041151    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6a47d53-96bf-459d-a3f6-0f8512c8732a-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:34.041178 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.041174    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glmxz\" (UniqueName: \"kubernetes.io/projected/e6a47d53-96bf-459d-a3f6-0f8512c8732a-kube-api-access-glmxz\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:34.041308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.041184    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-openshift-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:34.041308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.041193    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-metrics-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:34.041308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.041201    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-maas-api-tls\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:34.041308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.041208    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjqqq\" (UniqueName: \"kubernetes.io/projected/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a-kube-api-access-zjqqq\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:34.041308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.041217    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxxxv\" (UniqueName: \"kubernetes.io/projected/6e16b533-e89f-4c16-bd69-98e92d2ca43d-kube-api-access-gxxxv\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:34.270797 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.270760    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"]
Sep 19 15:42:34.275054 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.275026    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-88b9ff-5d964f9597-lgv59"]
Sep 19 15:42:34.283711 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.283685    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"]
Sep 19 15:42:34.287296 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.287275    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88b9ff-7c665ddb85-5fsd5"]
Sep 19 15:42:34.298465 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.298445    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"]
Sep 19 15:42:34.302571 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:34.302546    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-88b9ff-5df6f7df6b-wfl84"]
Sep 19 15:42:36.170702 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:36.170673    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e16b533-e89f-4c16-bd69-98e92d2ca43d" path="/var/lib/kubelet/pods/6e16b533-e89f-4c16-bd69-98e92d2ca43d/volumes"
Sep 19 15:42:36.171058 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:36.171038    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" path="/var/lib/kubelet/pods/8bed351e-cf1d-4900-b4e1-9d8e46f9d80a/volumes"
Sep 19 15:42:36.171357 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:36.171343    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6a47d53-96bf-459d-a3f6-0f8512c8732a" path="/var/lib/kubelet/pods/e6a47d53-96bf-459d-a3f6-0f8512c8732a/volumes"
Sep 19 15:42:56.505991 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.505959    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"]
Sep 19 15:42:56.508644 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.506199    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj" podUID="1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" containerName="maas-api" containerID="cri-o://2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0" gracePeriod=30
Sep 19 15:42:56.556247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.556219    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"]
Sep 19 15:42:56.556491 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.556442    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb" podUID="bf48da61-061b-4c6f-90b9-e05e1d3d2a0d" containerName="payload-processing" containerID="cri-o://674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f" gracePeriod=30
Sep 19 15:42:56.821983 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.821960    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:42:56.824647 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.824630    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:42:56.926774 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.926746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58k9j\" (UniqueName: \"kubernetes.io/projected/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-kube-api-access-58k9j\") pod \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\" (UID: \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\") "
Sep 19 15:42:56.926935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.926781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-metrics-certs\") pod \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") "
Sep 19 15:42:56.926935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.926834    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbfkk\" (UniqueName: \"kubernetes.io/projected/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-kube-api-access-hbfkk\") pod \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") "
Sep 19 15:42:56.926935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.926862    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-openshift-service-ca\") pod \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") "
Sep 19 15:42:56.926935 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.926904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-plugins-config-volume\") pod \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\" (UID: \"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d\") "
Sep 19 15:42:56.927135 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.926934    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls\") pod \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\" (UID: \"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87\") "
Sep 19 15:42:56.927290 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.927258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" (UID: "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:42:56.927408 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.927297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf48da61-061b-4c6f-90b9-e05e1d3d2a0d" (UID: "bf48da61-061b-4c6f-90b9-e05e1d3d2a0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:42:56.929156 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.929135    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-kube-api-access-hbfkk" (OuterVolumeSpecName: "kube-api-access-hbfkk") pod "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" (UID: "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87"). InnerVolumeSpecName "kube-api-access-hbfkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:42:56.929238 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.929168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" (UID: "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:42:56.929238 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.929175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-kube-api-access-58k9j" (OuterVolumeSpecName: "kube-api-access-58k9j") pod "bf48da61-061b-4c6f-90b9-e05e1d3d2a0d" (UID: "bf48da61-061b-4c6f-90b9-e05e1d3d2a0d"). InnerVolumeSpecName "kube-api-access-58k9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:42:56.929238 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:56.929181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" (UID: "1bd1c978-4b0d-4e92-95ab-4f81bdd5da87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:42:57.028142 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.028118    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbfkk\" (UniqueName: \"kubernetes.io/projected/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-kube-api-access-hbfkk\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:57.028142 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.028138    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-openshift-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:57.028269 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.028147    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:57.028269 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.028155    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-maas-api-tls\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:57.028269 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.028163    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58k9j\" (UniqueName: \"kubernetes.io/projected/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d-kube-api-access-58k9j\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:57.028269 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.028170    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87-metrics-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:42:57.058495 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.058437    2579 generic.go:356] "Generic (PLEG): container finished" podID="bf48da61-061b-4c6f-90b9-e05e1d3d2a0d" containerID="674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f" exitCode=0
Sep 19 15:42:57.058495 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.058483    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"
Sep 19 15:42:57.058698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.058534    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb" event={"ID":"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d","Type":"ContainerDied","Data":"674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f"}
Sep 19 15:42:57.058698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.058595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb" event={"ID":"bf48da61-061b-4c6f-90b9-e05e1d3d2a0d","Type":"ContainerDied","Data":"08b91079875f88767a7b8274da388c256014db1280cd3c899d5cfaf71b91c823"}
Sep 19 15:42:57.058698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.058658    2579 scope.go:117] "RemoveContainer" containerID="674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f"
Sep 19 15:42:57.060210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.060122    2579 generic.go:356] "Generic (PLEG): container finished" podID="1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" containerID="2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0" exitCode=0
Sep 19 15:42:57.060210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.060163    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"
Sep 19 15:42:57.060210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.060181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj" event={"ID":"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87","Type":"ContainerDied","Data":"2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0"}
Sep 19 15:42:57.060210 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.060216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj" event={"ID":"1bd1c978-4b0d-4e92-95ab-4f81bdd5da87","Type":"ContainerDied","Data":"0f687c7be4ef3da88217f3ca2aa541af278fc67bcc598a6642c8a9247d6487b8"}
Sep 19 15:42:57.067656 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.067590    2579 scope.go:117] "RemoveContainer" containerID="674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f"
Sep 19 15:42:57.067927 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:42:57.067906    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f\": container with ID starting with 674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f not found: ID does not exist" containerID="674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f"
Sep 19 15:42:57.068009 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.067936    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f"} err="failed to get container status \"674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f\": rpc error: code = NotFound desc = could not find container \"674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f\": container with ID starting with 674af684ec83e4ca0ec88a94b09c8075897ce634a2ff43246be35bac86e78e7f not found: ID does not exist"
Sep 19 15:42:57.068009 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.067952    2579 scope.go:117] "RemoveContainer" containerID="2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0"
Sep 19 15:42:57.075750 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.075731    2579 scope.go:117] "RemoveContainer" containerID="2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0"
Sep 19 15:42:57.075989 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:42:57.075972    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0\": container with ID starting with 2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0 not found: ID does not exist" containerID="2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0"
Sep 19 15:42:57.076035 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.075995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0"} err="failed to get container status \"2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0\": rpc error: code = NotFound desc = could not find container \"2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0\": container with ID starting with 2e0950252d4c1e3da38d9c81672e727dbaea13d18d66d685ada482d8c63dc8f0 not found: ID does not exist"
Sep 19 15:42:57.085017 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.084996    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"]
Sep 19 15:42:57.087785 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.087764    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26cbe6-6f86965f89-pkgrb"]
Sep 19 15:42:57.097448 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.097420    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"]
Sep 19 15:42:57.099223 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:57.099203    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj"]
Sep 19 15:42:58.170954 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:58.170925    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" path="/var/lib/kubelet/pods/1bd1c978-4b0d-4e92-95ab-4f81bdd5da87/volumes"
Sep 19 15:42:58.171321 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:58.171284    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf48da61-061b-4c6f-90b9-e05e1d3d2a0d" path="/var/lib/kubelet/pods/bf48da61-061b-4c6f-90b9-e05e1d3d2a0d/volumes"
Sep 19 15:42:59.728961 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:42:59.728921    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26cbe6-657854c945-826rj" podUID="1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.69:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 19 15:43:05.469065 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469034    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"]
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469352    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf48da61-061b-4c6f-90b9-e05e1d3d2a0d" containerName="payload-processing"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469363    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf48da61-061b-4c6f-90b9-e05e1d3d2a0d" containerName="payload-processing"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469373    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a47d53-96bf-459d-a3f6-0f8512c8732a" containerName="payload-pre-processing"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469379    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a47d53-96bf-459d-a3f6-0f8512c8732a" containerName="payload-pre-processing"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469388    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e16b533-e89f-4c16-bd69-98e92d2ca43d" containerName="payload-processing"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469395    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e16b533-e89f-4c16-bd69-98e92d2ca43d" containerName="payload-processing"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469407    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" containerName="maas-api"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469412    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" containerName="maas-api"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469429    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" containerName="maas-api"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469434    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" containerName="maas-api"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469485    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e16b533-e89f-4c16-bd69-98e92d2ca43d" containerName="payload-processing"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469492    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bed351e-cf1d-4900-b4e1-9d8e46f9d80a" containerName="maas-api"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469498    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf48da61-061b-4c6f-90b9-e05e1d3d2a0d" containerName="payload-processing"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469503    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd1c978-4b0d-4e92-95ab-4f81bdd5da87" containerName="maas-api"
Sep 19 15:43:05.469508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.469509    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6a47d53-96bf-459d-a3f6-0f8512c8732a" containerName="payload-pre-processing"
Sep 19 15:43:05.473813 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.473798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.476690 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.476669    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9c5c2\"/\"openshift-service-ca.crt\""
Sep 19 15:43:05.476824 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.476688    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9c5c2\"/\"default-dockercfg-zwxth\""
Sep 19 15:43:05.476824 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.476699    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9c5c2\"/\"rate-test-model-f9c5c2-kserve-self-signed-certs\""
Sep 19 15:43:05.476824 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.476743    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f9c5c2\"/\"kube-root-ca.crt\""
Sep 19 15:43:05.481944 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.481921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"]
Sep 19 15:43:05.602686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.602661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-model-cache\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.602824 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.602707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tls-certs\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.602824 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.602726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-dshm\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.602824 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.602801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-home\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.602925 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.602829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tmp-dir\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.602925 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.602880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tnz\" (UniqueName: \"kubernetes.io/projected/6ea2302b-0b69-4276-9b98-74be7d5d8a41-kube-api-access-99tnz\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.703727 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.703697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-dshm\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.703864 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.703729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-home\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.703864 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.703756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tmp-dir\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.703967 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.703875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tnz\" (UniqueName: \"kubernetes.io/projected/6ea2302b-0b69-4276-9b98-74be7d5d8a41-kube-api-access-99tnz\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.703967 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.703932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-model-cache\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.704122 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.704098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tmp-dir\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.704122 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.704117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tls-certs\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.704319 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.704122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-home\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.704319 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.704275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-model-cache\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.706156 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.706129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-dshm\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.706428 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.706413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tls-certs\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.711119 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.711103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tnz\" (UniqueName: \"kubernetes.io/projected/6ea2302b-0b69-4276-9b98-74be7d5d8a41-kube-api-access-99tnz\") pod \"rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") " pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:05.783969 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:05.783910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:06.054412 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:06.054384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"]
Sep 19 15:43:06.056424 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:43:06.056394    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ea2302b_0b69_4276_9b98_74be7d5d8a41.slice/crio-be3e7bcca89ca068abaff5b9e4f2f7d5e7d9eeae09b25a7f492a912d142247eb WatchSource:0}: Error finding container be3e7bcca89ca068abaff5b9e4f2f7d5e7d9eeae09b25a7f492a912d142247eb: Status 404 returned error can't find the container with id be3e7bcca89ca068abaff5b9e4f2f7d5e7d9eeae09b25a7f492a912d142247eb
Sep 19 15:43:06.107349 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:06.107323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b" event={"ID":"6ea2302b-0b69-4276-9b98-74be7d5d8a41","Type":"ContainerStarted","Data":"be3e7bcca89ca068abaff5b9e4f2f7d5e7d9eeae09b25a7f492a912d142247eb"}
Sep 19 15:43:07.113997 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:07.113959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b" event={"ID":"6ea2302b-0b69-4276-9b98-74be7d5d8a41","Type":"ContainerStarted","Data":"fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431"}
Sep 19 15:43:07.130402 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:07.130351    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b" podStartSLOduration=2.130337681 podStartE2EDuration="2.130337681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:43:07.127922815 +0000 UTC m=+1849.021486456" watchObservedRunningTime="2026-09-19 15:43:07.130337681 +0000 UTC m=+1849.023901321"
Sep 19 15:43:13.888958 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:13.888916    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"]
Sep 19 15:43:13.889624 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:13.889568    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4" podUID="cba2ceb3-0102-467e-8633-3c8a005a2bb4" containerName="payload-processing" containerID="cri-o://032376d85c414ed19a1aa80bf4c801394f243e822ac06225dcffb24b96d1c2d6" gracePeriod=30
Sep 19 15:43:13.891902 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:13.891871    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"]
Sep 19 15:43:13.892153 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:13.892118    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj" podUID="13ffa437-b79c-4f4f-a4ff-f47de85cff3c" containerName="payload-pre-processing" containerID="cri-o://80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc" gracePeriod=30
Sep 19 15:43:14.148015 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.147941    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:43:14.150758 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.150734    2579 generic.go:356] "Generic (PLEG): container finished" podID="13ffa437-b79c-4f4f-a4ff-f47de85cff3c" containerID="80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc" exitCode=0
Sep 19 15:43:14.150889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.150772    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"
Sep 19 15:43:14.150889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.150815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj" event={"ID":"13ffa437-b79c-4f4f-a4ff-f47de85cff3c","Type":"ContainerDied","Data":"80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc"}
Sep 19 15:43:14.150889 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.150867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj" event={"ID":"13ffa437-b79c-4f4f-a4ff-f47de85cff3c","Type":"ContainerDied","Data":"4fcef283c9a7d90bcb8ad7edc02a3453a14cd237bf5c92453825224e042bb370"}
Sep 19 15:43:14.151053 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.150893    2579 scope.go:117] "RemoveContainer" containerID="80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc"
Sep 19 15:43:14.152921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.152897    2579 generic.go:356] "Generic (PLEG): container finished" podID="cba2ceb3-0102-467e-8633-3c8a005a2bb4" containerID="032376d85c414ed19a1aa80bf4c801394f243e822ac06225dcffb24b96d1c2d6" exitCode=0
Sep 19 15:43:14.153036 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.152934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4" event={"ID":"cba2ceb3-0102-467e-8633-3c8a005a2bb4","Type":"ContainerDied","Data":"032376d85c414ed19a1aa80bf4c801394f243e822ac06225dcffb24b96d1c2d6"}
Sep 19 15:43:14.171526 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.171508    2579 scope.go:117] "RemoveContainer" containerID="80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc"
Sep 19 15:43:14.172086 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:43:14.172064    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc\": container with ID starting with 80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc not found: ID does not exist" containerID="80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc"
Sep 19 15:43:14.172222 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.172198    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc"} err="failed to get container status \"80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc\": rpc error: code = NotFound desc = could not find container \"80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc\": container with ID starting with 80ccdc8112968fc2db1a78b8701c66259d1d44769f603293f5da160026d999dc not found: ID does not exist"
Sep 19 15:43:14.175463 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.175443    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:43:14.279065 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.279040    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxfwc\" (UniqueName: \"kubernetes.io/projected/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-kube-api-access-jxfwc\") pod \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\" (UID: \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\") "
Sep 19 15:43:14.279177 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.279104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-plugins-config-volume\") pod \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\" (UID: \"13ffa437-b79c-4f4f-a4ff-f47de85cff3c\") "
Sep 19 15:43:14.279177 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.279127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba2ceb3-0102-467e-8633-3c8a005a2bb4-plugins-config-volume\") pod \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\" (UID: \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\") "
Sep 19 15:43:14.279177 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.279153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9g6l\" (UniqueName: \"kubernetes.io/projected/cba2ceb3-0102-467e-8633-3c8a005a2bb4-kube-api-access-p9g6l\") pod \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\" (UID: \"cba2ceb3-0102-467e-8633-3c8a005a2bb4\") "
Sep 19 15:43:14.279489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.279465    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13ffa437-b79c-4f4f-a4ff-f47de85cff3c" (UID: "13ffa437-b79c-4f4f-a4ff-f47de85cff3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:43:14.279552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.279523    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba2ceb3-0102-467e-8633-3c8a005a2bb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cba2ceb3-0102-467e-8633-3c8a005a2bb4" (UID: "cba2ceb3-0102-467e-8633-3c8a005a2bb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:43:14.281242 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.281204    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-kube-api-access-jxfwc" (OuterVolumeSpecName: "kube-api-access-jxfwc") pod "13ffa437-b79c-4f4f-a4ff-f47de85cff3c" (UID: "13ffa437-b79c-4f4f-a4ff-f47de85cff3c"). InnerVolumeSpecName "kube-api-access-jxfwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:43:14.281323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.281278    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba2ceb3-0102-467e-8633-3c8a005a2bb4-kube-api-access-p9g6l" (OuterVolumeSpecName: "kube-api-access-p9g6l") pod "cba2ceb3-0102-467e-8633-3c8a005a2bb4" (UID: "cba2ceb3-0102-467e-8633-3c8a005a2bb4"). InnerVolumeSpecName "kube-api-access-p9g6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:43:14.380700 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.380676    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxfwc\" (UniqueName: \"kubernetes.io/projected/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-kube-api-access-jxfwc\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:43:14.380700 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.380696    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ffa437-b79c-4f4f-a4ff-f47de85cff3c-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:43:14.380815 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.380706    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba2ceb3-0102-467e-8633-3c8a005a2bb4-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:43:14.380815 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.380715    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9g6l\" (UniqueName: \"kubernetes.io/projected/cba2ceb3-0102-467e-8633-3c8a005a2bb4-kube-api-access-p9g6l\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:43:14.471226 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.471205    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"]
Sep 19 15:43:14.474308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:14.474287    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c8e1cf-7488c79cf8-jplrj"]
Sep 19 15:43:15.158625 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:15.158532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"
Sep 19 15:43:15.158625 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:15.158537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4" event={"ID":"cba2ceb3-0102-467e-8633-3c8a005a2bb4","Type":"ContainerDied","Data":"444bf4891430a713f72e5f8db9b41042bb229c7819a527808008886010cfc98c"}
Sep 19 15:43:15.159109 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:15.158667    2579 scope.go:117] "RemoveContainer" containerID="032376d85c414ed19a1aa80bf4c801394f243e822ac06225dcffb24b96d1c2d6"
Sep 19 15:43:15.178133 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:15.178109    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"]
Sep 19 15:43:15.192738 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:15.192716    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c8e1cf-6848bf5687-4w2d4"]
Sep 19 15:43:15.784008 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:15.783978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:15.784178 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:15.784020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:15.795962 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:15.795941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:16.170475 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:16.170403    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13ffa437-b79c-4f4f-a4ff-f47de85cff3c" path="/var/lib/kubelet/pods/13ffa437-b79c-4f4f-a4ff-f47de85cff3c/volumes"
Sep 19 15:43:16.170873 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:16.170768    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba2ceb3-0102-467e-8633-3c8a005a2bb4" path="/var/lib/kubelet/pods/cba2ceb3-0102-467e-8633-3c8a005a2bb4/volumes"
Sep 19 15:43:16.176592 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:16.176569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:43:31.289995 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.289921    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"]
Sep 19 15:43:31.290479 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.290163    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq" podUID="918c32a2-a1a7-43d6-b59c-72c3c4a26fd0" containerName="payload-processing" containerID="cri-o://e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1" gracePeriod=30
Sep 19 15:43:31.292909 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.292883    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"]
Sep 19 15:43:31.294043 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.293998    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf" podUID="8d13c49b-6b40-42bd-8a6a-91ef82f961c7" containerName="payload-pre-processing" containerID="cri-o://dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660" gracePeriod=30
Sep 19 15:43:31.563237 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.563216    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:43:31.565773 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.565755    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:43:31.718275 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.718238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-plugins-config-volume\") pod \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\" (UID: \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\") "
Sep 19 15:43:31.718427 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.718304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r426k\" (UniqueName: \"kubernetes.io/projected/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-kube-api-access-r426k\") pod \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\" (UID: \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\") "
Sep 19 15:43:31.718427 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.718357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xkn5\" (UniqueName: \"kubernetes.io/projected/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-kube-api-access-2xkn5\") pod \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\" (UID: \"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0\") "
Sep 19 15:43:31.718427 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.718381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-plugins-config-volume\") pod \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\" (UID: \"8d13c49b-6b40-42bd-8a6a-91ef82f961c7\") "
Sep 19 15:43:31.718572 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.718549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "918c32a2-a1a7-43d6-b59c-72c3c4a26fd0" (UID: "918c32a2-a1a7-43d6-b59c-72c3c4a26fd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:43:31.718666 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.718643    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:43:31.718820 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.718800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d13c49b-6b40-42bd-8a6a-91ef82f961c7" (UID: "8d13c49b-6b40-42bd-8a6a-91ef82f961c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:43:31.720500 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.720483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-kube-api-access-r426k" (OuterVolumeSpecName: "kube-api-access-r426k") pod "8d13c49b-6b40-42bd-8a6a-91ef82f961c7" (UID: "8d13c49b-6b40-42bd-8a6a-91ef82f961c7"). InnerVolumeSpecName "kube-api-access-r426k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:43:31.720544 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.720527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-kube-api-access-2xkn5" (OuterVolumeSpecName: "kube-api-access-2xkn5") pod "918c32a2-a1a7-43d6-b59c-72c3c4a26fd0" (UID: "918c32a2-a1a7-43d6-b59c-72c3c4a26fd0"). InnerVolumeSpecName "kube-api-access-2xkn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:43:31.819509 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.819452    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xkn5\" (UniqueName: \"kubernetes.io/projected/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0-kube-api-access-2xkn5\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:43:31.819509 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.819474    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:43:31.819509 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:31.819483    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r426k\" (UniqueName: \"kubernetes.io/projected/8d13c49b-6b40-42bd-8a6a-91ef82f961c7-kube-api-access-r426k\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:43:32.235536 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.235513    2579 generic.go:356] "Generic (PLEG): container finished" podID="918c32a2-a1a7-43d6-b59c-72c3c4a26fd0" containerID="e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1" exitCode=0
Sep 19 15:43:32.235671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.235551    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"
Sep 19 15:43:32.235671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.235596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq" event={"ID":"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0","Type":"ContainerDied","Data":"e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1"}
Sep 19 15:43:32.235671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.235668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq" event={"ID":"918c32a2-a1a7-43d6-b59c-72c3c4a26fd0","Type":"ContainerDied","Data":"76cc9f30c160a7dedc0b0b923e79812f89a7326782635a93a8dc42cfe1ec6009"}
Sep 19 15:43:32.235828 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.235695    2579 scope.go:117] "RemoveContainer" containerID="e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1"
Sep 19 15:43:32.237030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.236997    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d13c49b-6b40-42bd-8a6a-91ef82f961c7" containerID="dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660" exitCode=0
Sep 19 15:43:32.237137 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.237052    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"
Sep 19 15:43:32.237137 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.237063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf" event={"ID":"8d13c49b-6b40-42bd-8a6a-91ef82f961c7","Type":"ContainerDied","Data":"dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660"}
Sep 19 15:43:32.237137 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.237103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf" event={"ID":"8d13c49b-6b40-42bd-8a6a-91ef82f961c7","Type":"ContainerDied","Data":"1ed465aab774d1fb67af9b851e15d99a79b120e1946fde4e83a7209bf2a51974"}
Sep 19 15:43:32.243853 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.243838    2579 scope.go:117] "RemoveContainer" containerID="e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1"
Sep 19 15:43:32.244130 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:43:32.244108    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1\": container with ID starting with e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1 not found: ID does not exist" containerID="e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1"
Sep 19 15:43:32.244205 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.244142    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1"} err="failed to get container status \"e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1\": rpc error: code = NotFound desc = could not find container \"e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1\": container with ID starting with e9a7554cb00b4b45effe6dc7525779a6734378d6411ebd298dc2f30ae24fd6c1 not found: ID does not exist"
Sep 19 15:43:32.244205 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.244163    2579 scope.go:117] "RemoveContainer" containerID="dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660"
Sep 19 15:43:32.250517 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.250498    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"]
Sep 19 15:43:32.251980 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.251963    2579 scope.go:117] "RemoveContainer" containerID="dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660"
Sep 19 15:43:32.252218 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:43:32.252203    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660\": container with ID starting with dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660 not found: ID does not exist" containerID="dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660"
Sep 19 15:43:32.252274 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.252225    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660"} err="failed to get container status \"dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660\": rpc error: code = NotFound desc = could not find container \"dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660\": container with ID starting with dc2f0d2693942ce9b6f2dad5e61f66559f06e0312954c689afc7912f74980660 not found: ID does not exist"
Sep 19 15:43:32.256367 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.256338    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-da587f-5cf56fcf5f-gn2sq"]
Sep 19 15:43:32.267171 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.267147    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"]
Sep 19 15:43:32.276833 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:32.276814    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-da587f-6d788dc4f8-d2ppf"]
Sep 19 15:43:34.170733 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:34.170705    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d13c49b-6b40-42bd-8a6a-91ef82f961c7" path="/var/lib/kubelet/pods/8d13c49b-6b40-42bd-8a6a-91ef82f961c7/volumes"
Sep 19 15:43:34.171105 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:34.171042    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="918c32a2-a1a7-43d6-b59c-72c3c4a26fd0" path="/var/lib/kubelet/pods/918c32a2-a1a7-43d6-b59c-72c3c4a26fd0/volumes"
Sep 19 15:43:55.013254 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:43:55.013217    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"]
Sep 19 15:44:00.440473 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.440437    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"]
Sep 19 15:44:00.440946 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.440924    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918c32a2-a1a7-43d6-b59c-72c3c4a26fd0" containerName="payload-processing"
Sep 19 15:44:00.441012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.440950    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="918c32a2-a1a7-43d6-b59c-72c3c4a26fd0" containerName="payload-processing"
Sep 19 15:44:00.441012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.440968    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ffa437-b79c-4f4f-a4ff-f47de85cff3c" containerName="payload-pre-processing"
Sep 19 15:44:00.441012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.440977    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ffa437-b79c-4f4f-a4ff-f47de85cff3c" containerName="payload-pre-processing"
Sep 19 15:44:00.441012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.440998    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba2ceb3-0102-467e-8633-3c8a005a2bb4" containerName="payload-processing"
Sep 19 15:44:00.441012 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.441008    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba2ceb3-0102-467e-8633-3c8a005a2bb4" containerName="payload-processing"
Sep 19 15:44:00.441163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.441029    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d13c49b-6b40-42bd-8a6a-91ef82f961c7" containerName="payload-pre-processing"
Sep 19 15:44:00.441163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.441038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d13c49b-6b40-42bd-8a6a-91ef82f961c7" containerName="payload-pre-processing"
Sep 19 15:44:00.441163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.441121    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ffa437-b79c-4f4f-a4ff-f47de85cff3c" containerName="payload-pre-processing"
Sep 19 15:44:00.441163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.441136    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba2ceb3-0102-467e-8633-3c8a005a2bb4" containerName="payload-processing"
Sep 19 15:44:00.441163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.441147    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="918c32a2-a1a7-43d6-b59c-72c3c4a26fd0" containerName="payload-processing"
Sep 19 15:44:00.441163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.441158    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d13c49b-6b40-42bd-8a6a-91ef82f961c7" containerName="payload-pre-processing"
Sep 19 15:44:00.444458 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.444437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.447193 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.447142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 15:44:00.447318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.447189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-521326-serving-cert\""
Sep 19 15:44:00.447460 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.447441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 15:44:00.447528 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.447443    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zhfqt\""
Sep 19 15:44:00.447528 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.447501    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 15:44:00.453031 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.452563    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"]
Sep 19 15:44:00.525871 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.525842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-metrics-certs\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.526007 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.525875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgld\" (UniqueName: \"kubernetes.io/projected/8a8874df-a4c6-448b-a9ea-41d4c863facc-kube-api-access-mlgld\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.526007 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.525992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8874df-a4c6-448b-a9ea-41d4c863facc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.526106 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.526016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.626796 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.626776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8874df-a4c6-448b-a9ea-41d4c863facc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.626940 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.626801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.626940 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.626846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-metrics-certs\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.626940 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.626863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgld\" (UniqueName: \"kubernetes.io/projected/8a8874df-a4c6-448b-a9ea-41d4c863facc-kube-api-access-mlgld\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.627452 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.627434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8874df-a4c6-448b-a9ea-41d4c863facc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.629584 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.629560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-metrics-certs\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.629758 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.629741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.636304 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.636281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgld\" (UniqueName: \"kubernetes.io/projected/8a8874df-a4c6-448b-a9ea-41d4c863facc-kube-api-access-mlgld\") pod \"maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:00.758974 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:00.758920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:01.033806 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.033731    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"]
Sep 19 15:44:01.036957 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:44:01.036933    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8874df_a4c6_448b_a9ea_41d4c863facc.slice/crio-506af70200d2bf87375d9fc4b7ebe2396e6cab57c3e1cac033fe8fe9d825c6d6 WatchSource:0}: Error finding container 506af70200d2bf87375d9fc4b7ebe2396e6cab57c3e1cac033fe8fe9d825c6d6: Status 404 returned error can't find the container with id 506af70200d2bf87375d9fc4b7ebe2396e6cab57c3e1cac033fe8fe9d825c6d6
Sep 19 15:44:01.056770 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.056740    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"]
Sep 19 15:44:01.061370 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.061352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:01.063357 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.063341    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-521326-dockercfg-94zxh\""
Sep 19 15:44:01.063955 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.063935    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-521326\""
Sep 19 15:44:01.070305 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.070286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"]
Sep 19 15:44:01.073762 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.073741    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"]
Sep 19 15:44:01.076768 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.076748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:01.088733 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.088710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"]
Sep 19 15:44:01.133649 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.133627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5mv\" (UniqueName: \"kubernetes.io/projected/c5dee583-72f2-4408-906c-338543636c49-kube-api-access-mv5mv\") pod \"payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn\" (UID: \"c5dee583-72f2-4408-906c-338543636c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:01.133757 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.133667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t4x\" (UniqueName: \"kubernetes.io/projected/8b1367ca-56c9-4259-bfb7-4653908738d9-kube-api-access-b6t4x\") pod \"payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv\" (UID: \"8b1367ca-56c9-4259-bfb7-4653908738d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:01.133807 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.133752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dee583-72f2-4408-906c-338543636c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn\" (UID: \"c5dee583-72f2-4408-906c-338543636c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:01.133807 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.133778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1367ca-56c9-4259-bfb7-4653908738d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv\" (UID: \"8b1367ca-56c9-4259-bfb7-4653908738d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:01.235029 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.234998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5mv\" (UniqueName: \"kubernetes.io/projected/c5dee583-72f2-4408-906c-338543636c49-kube-api-access-mv5mv\") pod \"payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn\" (UID: \"c5dee583-72f2-4408-906c-338543636c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:01.235160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.235046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t4x\" (UniqueName: \"kubernetes.io/projected/8b1367ca-56c9-4259-bfb7-4653908738d9-kube-api-access-b6t4x\") pod \"payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv\" (UID: \"8b1367ca-56c9-4259-bfb7-4653908738d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:01.235160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.235113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dee583-72f2-4408-906c-338543636c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn\" (UID: \"c5dee583-72f2-4408-906c-338543636c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:01.235160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.235140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1367ca-56c9-4259-bfb7-4653908738d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv\" (UID: \"8b1367ca-56c9-4259-bfb7-4653908738d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:01.235781 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.235761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dee583-72f2-4408-906c-338543636c49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn\" (UID: \"c5dee583-72f2-4408-906c-338543636c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:01.235905 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.235885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1367ca-56c9-4259-bfb7-4653908738d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv\" (UID: \"8b1367ca-56c9-4259-bfb7-4653908738d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:01.242409 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.242391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5mv\" (UniqueName: \"kubernetes.io/projected/c5dee583-72f2-4408-906c-338543636c49-kube-api-access-mv5mv\") pod \"payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn\" (UID: \"c5dee583-72f2-4408-906c-338543636c49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:01.242500 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.242426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t4x\" (UniqueName: \"kubernetes.io/projected/8b1367ca-56c9-4259-bfb7-4653908738d9-kube-api-access-b6t4x\") pod \"payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv\" (UID: \"8b1367ca-56c9-4259-bfb7-4653908738d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:01.363530 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.363475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67" event={"ID":"8a8874df-a4c6-448b-a9ea-41d4c863facc","Type":"ContainerStarted","Data":"506af70200d2bf87375d9fc4b7ebe2396e6cab57c3e1cac033fe8fe9d825c6d6"}
Sep 19 15:44:01.373619 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.373582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:01.385468 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.385448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:01.643489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.643449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"]
Sep 19 15:44:01.644735 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:44:01.644709    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5dee583_72f2_4408_906c_338543636c49.slice/crio-54a362ad911ed8ca68079ec525c3df10497bbdee57faa6193befa8636417fb38 WatchSource:0}: Error finding container 54a362ad911ed8ca68079ec525c3df10497bbdee57faa6193befa8636417fb38: Status 404 returned error can't find the container with id 54a362ad911ed8ca68079ec525c3df10497bbdee57faa6193befa8636417fb38
Sep 19 15:44:01.660963 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:01.660945    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"]
Sep 19 15:44:01.662563 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:44:01.662540    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1367ca_56c9_4259_bfb7_4653908738d9.slice/crio-373a9321337c3be6a0515e1a3c202e3b8e237374fc33cc089ea0106f8b342b44 WatchSource:0}: Error finding container 373a9321337c3be6a0515e1a3c202e3b8e237374fc33cc089ea0106f8b342b44: Status 404 returned error can't find the container with id 373a9321337c3be6a0515e1a3c202e3b8e237374fc33cc089ea0106f8b342b44
Sep 19 15:44:02.370289 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.370251    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn" event={"ID":"c5dee583-72f2-4408-906c-338543636c49","Type":"ContainerStarted","Data":"fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35"}
Sep 19 15:44:02.370489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.370440    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:02.370489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.370482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn" event={"ID":"c5dee583-72f2-4408-906c-338543636c49","Type":"ContainerStarted","Data":"54a362ad911ed8ca68079ec525c3df10497bbdee57faa6193befa8636417fb38"}
Sep 19 15:44:02.372059 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.372019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv" event={"ID":"8b1367ca-56c9-4259-bfb7-4653908738d9","Type":"ContainerStarted","Data":"ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71"}
Sep 19 15:44:02.372059 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.372060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv" event={"ID":"8b1367ca-56c9-4259-bfb7-4653908738d9","Type":"ContainerStarted","Data":"373a9321337c3be6a0515e1a3c202e3b8e237374fc33cc089ea0106f8b342b44"}
Sep 19 15:44:02.372263 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.372141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:02.373575 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.373555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67" event={"ID":"8a8874df-a4c6-448b-a9ea-41d4c863facc","Type":"ContainerStarted","Data":"88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285"}
Sep 19 15:44:02.373723 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.373657    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:02.384736 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.384696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn" podStartSLOduration=1.384684159 podStartE2EDuration="1.384684159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:44:02.383211235 +0000 UTC m=+1904.276774852" watchObservedRunningTime="2026-09-19 15:44:02.384684159 +0000 UTC m=+1904.278247798"
Sep 19 15:44:02.399485 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.399436    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67" podStartSLOduration=1.929126476 podStartE2EDuration="2.399420037s" totalImagesPullingTime="470.293561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:44:02.398075546 +0000 UTC m=+1904.291639185" watchObservedRunningTime="2026-09-19 15:44:02.399420037 +0000 UTC m=+1904.292983682"
Sep 19 15:44:02.413098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:02.413055    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv" podStartSLOduration=1.413039894 podStartE2EDuration="1.413039894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:44:02.412103627 +0000 UTC m=+1904.305667267" watchObservedRunningTime="2026-09-19 15:44:02.413039894 +0000 UTC m=+1904.306603540"
Sep 19 15:44:08.384663 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:08.384569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:44:10.033480 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.033441    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b" podUID="6ea2302b-0b69-4276-9b98-74be7d5d8a41" containerName="main" containerID="cri-o://fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431" gracePeriod=45
Sep 19 15:44:10.276750 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.276725    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:44:10.407508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.407454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-dshm\") pod \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") "
Sep 19 15:44:10.407508 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.407496    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-model-cache\") pod \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") "
Sep 19 15:44:10.407686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.407521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-home\") pod \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") "
Sep 19 15:44:10.407686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.407628    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tls-certs\") pod \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") "
Sep 19 15:44:10.407686 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.407658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99tnz\" (UniqueName: \"kubernetes.io/projected/6ea2302b-0b69-4276-9b98-74be7d5d8a41-kube-api-access-99tnz\") pod \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") "
Sep 19 15:44:10.407843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.407708    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tmp-dir\") pod \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\" (UID: \"6ea2302b-0b69-4276-9b98-74be7d5d8a41\") "
Sep 19 15:44:10.407843 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.407791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ea2302b-0b69-4276-9b98-74be7d5d8a41" (UID: "6ea2302b-0b69-4276-9b98-74be7d5d8a41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:44:10.407980 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.407877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-home" (OuterVolumeSpecName: "home") pod "6ea2302b-0b69-4276-9b98-74be7d5d8a41" (UID: "6ea2302b-0b69-4276-9b98-74be7d5d8a41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:44:10.408049 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.408030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ea2302b-0b69-4276-9b98-74be7d5d8a41" (UID: "6ea2302b-0b69-4276-9b98-74be7d5d8a41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:44:10.408123 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.408105    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:44:10.408180 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.408127    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:44:10.409760 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.409734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-dshm" (OuterVolumeSpecName: "dshm") pod "6ea2302b-0b69-4276-9b98-74be7d5d8a41" (UID: "6ea2302b-0b69-4276-9b98-74be7d5d8a41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:44:10.409881 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.409861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ea2302b-0b69-4276-9b98-74be7d5d8a41" (UID: "6ea2302b-0b69-4276-9b98-74be7d5d8a41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:44:10.410140 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.410113    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea2302b-0b69-4276-9b98-74be7d5d8a41-kube-api-access-99tnz" (OuterVolumeSpecName: "kube-api-access-99tnz") pod "6ea2302b-0b69-4276-9b98-74be7d5d8a41" (UID: "6ea2302b-0b69-4276-9b98-74be7d5d8a41"). InnerVolumeSpecName "kube-api-access-99tnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:44:10.413393 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.413369    2579 generic.go:356] "Generic (PLEG): container finished" podID="6ea2302b-0b69-4276-9b98-74be7d5d8a41" containerID="fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431" exitCode=0
Sep 19 15:44:10.413469 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.413406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b" event={"ID":"6ea2302b-0b69-4276-9b98-74be7d5d8a41","Type":"ContainerDied","Data":"fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431"}
Sep 19 15:44:10.413469 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.413429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b" event={"ID":"6ea2302b-0b69-4276-9b98-74be7d5d8a41","Type":"ContainerDied","Data":"be3e7bcca89ca068abaff5b9e4f2f7d5e7d9eeae09b25a7f492a912d142247eb"}
Sep 19 15:44:10.413469 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.413407    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"
Sep 19 15:44:10.413469 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.413445    2579 scope.go:117] "RemoveContainer" containerID="fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431"
Sep 19 15:44:10.429640 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.429597    2579 scope.go:117] "RemoveContainer" containerID="fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431"
Sep 19 15:44:10.429928 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:44:10.429908    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431\": container with ID starting with fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431 not found: ID does not exist" containerID="fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431"
Sep 19 15:44:10.429996 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.429937    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431"} err="failed to get container status \"fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431\": rpc error: code = NotFound desc = could not find container \"fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431\": container with ID starting with fffef3f8393deb05f56cfb60e497afd60ec8dcd5e192546a50411e3cb194c431 not found: ID does not exist"
Sep 19 15:44:10.439574 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.439551    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"]
Sep 19 15:44:10.443314 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.443293    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f9c5c2/rate-test-model-f9c5c2-kserve-65d8bdddf5-4qw7b"]
Sep 19 15:44:10.508641 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.508594    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:44:10.508719 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.508647    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea2302b-0b69-4276-9b98-74be7d5d8a41-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:44:10.508719 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.508659    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea2302b-0b69-4276-9b98-74be7d5d8a41-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:44:10.508719 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:10.508671    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tnz\" (UniqueName: \"kubernetes.io/projected/6ea2302b-0b69-4276-9b98-74be7d5d8a41-kube-api-access-99tnz\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:44:12.169978 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:12.169945    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea2302b-0b69-4276-9b98-74be7d5d8a41" path="/var/lib/kubelet/pods/6ea2302b-0b69-4276-9b98-74be7d5d8a41/volumes"
Sep 19 15:44:13.381533 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:13.381506    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:44:13.381907 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:13.381806    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:44:30.546285 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.546251    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"]
Sep 19 15:44:30.546742 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.546588    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea2302b-0b69-4276-9b98-74be7d5d8a41" containerName="main"
Sep 19 15:44:30.546742 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.546628    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea2302b-0b69-4276-9b98-74be7d5d8a41" containerName="main"
Sep 19 15:44:30.546742 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.546700    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea2302b-0b69-4276-9b98-74be7d5d8a41" containerName="main"
Sep 19 15:44:30.551203 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.551175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.553242 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.553216    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-30ef69-serving-cert\""
Sep 19 15:44:30.559056 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.559031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"]
Sep 19 15:44:30.660660 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.660637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.660775 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.660709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cad06f-a1a9-4021-a803-5757185d6d06-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.660775 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.660730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqrq\" (UniqueName: \"kubernetes.io/projected/a8cad06f-a1a9-4021-a803-5757185d6d06-kube-api-access-7qqrq\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.660775 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.660746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-metrics-certs\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.761950 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.761927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cad06f-a1a9-4021-a803-5757185d6d06-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.762067 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.761959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqrq\" (UniqueName: \"kubernetes.io/projected/a8cad06f-a1a9-4021-a803-5757185d6d06-kube-api-access-7qqrq\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.762067 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.761979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-metrics-certs\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.762067 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.762010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.762234 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:44:30.762108    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-serving-cert: secret "maas-api-e2e-ipp-b-30ef69-serving-cert" not found
Sep 19 15:44:30.762234 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:44:30.762176    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls podName:a8cad06f-a1a9-4021-a803-5757185d6d06 nodeName:}" failed. No retries permitted until 2026-09-19 15:44:31.262157273 +0000 UTC m=+1933.155720893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls") pod "maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh" (UID: "a8cad06f-a1a9-4021-a803-5757185d6d06") : secret "maas-api-e2e-ipp-b-30ef69-serving-cert" not found
Sep 19 15:44:30.762643 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.762591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cad06f-a1a9-4021-a803-5757185d6d06-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.764525 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.764496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-metrics-certs\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:30.772552 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:30.772529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqrq\" (UniqueName: \"kubernetes.io/projected/a8cad06f-a1a9-4021-a803-5757185d6d06-kube-api-access-7qqrq\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:31.245457 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.245425    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"]
Sep 19 15:44:31.248961 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.248943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:31.251070 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.251044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30ef69\""
Sep 19 15:44:31.251177 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.251096    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30ef69-dockercfg-m8k8g\""
Sep 19 15:44:31.257724 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.257704    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"]
Sep 19 15:44:31.265822 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.265800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:31.268267 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.268248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:31.282431 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.282405    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"]
Sep 19 15:44:31.285574 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.285559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:31.298462 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.298439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"]
Sep 19 15:44:31.366886 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.366856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw\" (UID: \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:31.367022 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.366940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbz5l\" (UniqueName: \"kubernetes.io/projected/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-kube-api-access-rbz5l\") pod \"payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw\" (UID: \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:31.367022 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.366987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39b3662-9bfb-437e-833f-d9d1e2097eda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6\" (UID: \"f39b3662-9bfb-437e-833f-d9d1e2097eda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:31.367112 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.367032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkvb\" (UniqueName: \"kubernetes.io/projected/f39b3662-9bfb-437e-833f-d9d1e2097eda-kube-api-access-tpkvb\") pod \"payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6\" (UID: \"f39b3662-9bfb-437e-833f-d9d1e2097eda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:31.464419 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.464395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:31.468401 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.468380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw\" (UID: \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:31.468506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.468441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbz5l\" (UniqueName: \"kubernetes.io/projected/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-kube-api-access-rbz5l\") pod \"payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw\" (UID: \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:31.468506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.468491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39b3662-9bfb-437e-833f-d9d1e2097eda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6\" (UID: \"f39b3662-9bfb-437e-833f-d9d1e2097eda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:31.468645 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.468526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkvb\" (UniqueName: \"kubernetes.io/projected/f39b3662-9bfb-437e-833f-d9d1e2097eda-kube-api-access-tpkvb\") pod \"payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6\" (UID: \"f39b3662-9bfb-437e-833f-d9d1e2097eda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:31.469258 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.469236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw\" (UID: \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:31.469316 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.469292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39b3662-9bfb-437e-833f-d9d1e2097eda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6\" (UID: \"f39b3662-9bfb-437e-833f-d9d1e2097eda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:31.476223 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.476197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbz5l\" (UniqueName: \"kubernetes.io/projected/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-kube-api-access-rbz5l\") pod \"payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw\" (UID: \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:31.476391 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.476372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkvb\" (UniqueName: \"kubernetes.io/projected/f39b3662-9bfb-437e-833f-d9d1e2097eda-kube-api-access-tpkvb\") pod \"payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6\" (UID: \"f39b3662-9bfb-437e-833f-d9d1e2097eda\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:31.560032 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.559998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:31.595391 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.595357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:31.732530 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.732505    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"]
Sep 19 15:44:31.733829 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:44:31.733804    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cad06f_a1a9_4021_a803_5757185d6d06.slice/crio-a14e3937d1748f59ae288780fab51f653f5dea235b909eccf8cf3b17de206bc2 WatchSource:0}: Error finding container a14e3937d1748f59ae288780fab51f653f5dea235b909eccf8cf3b17de206bc2: Status 404 returned error can't find the container with id a14e3937d1748f59ae288780fab51f653f5dea235b909eccf8cf3b17de206bc2
Sep 19 15:44:31.841241 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.841219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"]
Sep 19 15:44:31.842170 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:44:31.842148    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4768cb4_1f3f_49d5_9f51_cfdefee28ae6.slice/crio-1802b6042470bc06fb54b8aa27a2604729ca3f4d6d928ccafbed0fdc7de835c8 WatchSource:0}: Error finding container 1802b6042470bc06fb54b8aa27a2604729ca3f4d6d928ccafbed0fdc7de835c8: Status 404 returned error can't find the container with id 1802b6042470bc06fb54b8aa27a2604729ca3f4d6d928ccafbed0fdc7de835c8
Sep 19 15:44:31.871426 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:31.871406    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"]
Sep 19 15:44:31.881558 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:44:31.881521    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39b3662_9bfb_437e_833f_d9d1e2097eda.slice/crio-63dc0f8b02d68b96fe932c401b189971b4af1f28098ea5cc33e3b9ab7e4161d5 WatchSource:0}: Error finding container 63dc0f8b02d68b96fe932c401b189971b4af1f28098ea5cc33e3b9ab7e4161d5: Status 404 returned error can't find the container with id 63dc0f8b02d68b96fe932c401b189971b4af1f28098ea5cc33e3b9ab7e4161d5
Sep 19 15:44:32.514335 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.514249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh" event={"ID":"a8cad06f-a1a9-4021-a803-5757185d6d06","Type":"ContainerStarted","Data":"62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1"}
Sep 19 15:44:32.514484 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.514342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:32.514484 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.514368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh" event={"ID":"a8cad06f-a1a9-4021-a803-5757185d6d06","Type":"ContainerStarted","Data":"a14e3937d1748f59ae288780fab51f653f5dea235b909eccf8cf3b17de206bc2"}
Sep 19 15:44:32.515851 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.515818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6" event={"ID":"f39b3662-9bfb-437e-833f-d9d1e2097eda","Type":"ContainerStarted","Data":"83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013"}
Sep 19 15:44:32.515966 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.515859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6" event={"ID":"f39b3662-9bfb-437e-833f-d9d1e2097eda","Type":"ContainerStarted","Data":"63dc0f8b02d68b96fe932c401b189971b4af1f28098ea5cc33e3b9ab7e4161d5"}
Sep 19 15:44:32.515966 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.515885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:32.517404 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.517381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw" event={"ID":"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6","Type":"ContainerStarted","Data":"c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387"}
Sep 19 15:44:32.517517 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.517417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw" event={"ID":"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6","Type":"ContainerStarted","Data":"1802b6042470bc06fb54b8aa27a2604729ca3f4d6d928ccafbed0fdc7de835c8"}
Sep 19 15:44:32.517517 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.517437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:32.530460 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.530422    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh" podStartSLOduration=2.068150157 podStartE2EDuration="2.530408828s" totalImagesPullingTime="462.258671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:44:32.529486949 +0000 UTC m=+1934.423050588" watchObservedRunningTime="2026-09-19 15:44:32.530408828 +0000 UTC m=+1934.423972528"
Sep 19 15:44:32.544942 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.544897    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw" podStartSLOduration=1.54488458 podStartE2EDuration="1.54488458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:44:32.54388057 +0000 UTC m=+1934.437444209" watchObservedRunningTime="2026-09-19 15:44:32.54488458 +0000 UTC m=+1934.438448219"
Sep 19 15:44:32.559759 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:32.559713    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6" podStartSLOduration=1.559700817 podStartE2EDuration="1.559700817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:44:32.557932549 +0000 UTC m=+1934.451496200" watchObservedRunningTime="2026-09-19 15:44:32.559700817 +0000 UTC m=+1934.453264456"
Sep 19 15:44:38.527388 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:38.527352    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:44:43.523765 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:43.523731    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:44:43.524205 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:43.523881    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:44:58.457406 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:44:58.457371    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6_f39b3662-9bfb-437e-833f-d9d1e2097eda/payload-processing/0.log"
Sep 19 15:45:00.135583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:00.135544    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"]
Sep 19 15:45:00.139295 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:00.139272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"
Sep 19 15:45:00.141754 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:00.141729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7znsl\""
Sep 19 15:45:00.146751 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:00.146727    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"]
Sep 19 15:45:00.152817 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:00.152798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"
Sep 19 15:45:00.422769 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:00.422734    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"]
Sep 19 15:45:00.425093 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:45:00.425065    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8af884a_4516_41ed_81c5_4056ca83452d.slice/crio-c7bc7e2ef6b292c15d0d331b861f3ea0a28f2b16c3c9919965b0e638b1dc77be WatchSource:0}: Error finding container c7bc7e2ef6b292c15d0d331b861f3ea0a28f2b16c3c9919965b0e638b1dc77be: Status 404 returned error can't find the container with id c7bc7e2ef6b292c15d0d331b861f3ea0a28f2b16c3c9919965b0e638b1dc77be
Sep 19 15:45:00.647639 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:00.647586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8" event={"ID":"d8af884a-4516-41ed-81c5-4056ca83452d","Type":"ContainerStarted","Data":"c7bc7e2ef6b292c15d0d331b861f3ea0a28f2b16c3c9919965b0e638b1dc77be"}
Sep 19 15:45:01.653787 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:01.653706    2579 generic.go:356] "Generic (PLEG): container finished" podID="d8af884a-4516-41ed-81c5-4056ca83452d" containerID="b1a9c5d630d29053f2e0fe5afc3de7d718afccea5f30103d0d2c2d0a01b7659b" exitCode=0
Sep 19 15:45:01.653787 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:01.653763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8" event={"ID":"d8af884a-4516-41ed-81c5-4056ca83452d","Type":"ContainerDied","Data":"b1a9c5d630d29053f2e0fe5afc3de7d718afccea5f30103d0d2c2d0a01b7659b"}
Sep 19 15:45:02.784135 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:02.784105    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"
Sep 19 15:45:03.664048 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:03.664021    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"
Sep 19 15:45:03.664206 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:03.664066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8" event={"ID":"d8af884a-4516-41ed-81c5-4056ca83452d","Type":"ContainerDied","Data":"c7bc7e2ef6b292c15d0d331b861f3ea0a28f2b16c3c9919965b0e638b1dc77be"}
Sep 19 15:45:03.664206 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:45:03.664110    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7bc7e2ef6b292c15d0d331b861f3ea0a28f2b16c3c9919965b0e638b1dc77be"
Sep 19 15:46:00.065303 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:00.065274    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv_8b1367ca-56c9-4259-bfb7-4653908738d9/payload-processing/0.log"
Sep 19 15:46:00.415340 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:00.415265    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6_f39b3662-9bfb-437e-833f-d9d1e2097eda/payload-processing/0.log"
Sep 19 15:46:00.535556 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:00.535533    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6_f39b3662-9bfb-437e-833f-d9d1e2097eda/payload-processing/0.log"
Sep 19 15:46:07.411560 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.411522    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"]
Sep 19 15:46:07.412084 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.412064    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8af884a-4516-41ed-81c5-4056ca83452d" containerName="cleanup"
Sep 19 15:46:07.412163 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.412087    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8af884a-4516-41ed-81c5-4056ca83452d" containerName="cleanup"
Sep 19 15:46:07.412217 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.412195    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8af884a-4516-41ed-81c5-4056ca83452d" containerName="cleanup"
Sep 19 15:46:07.415297 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.415276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:07.417537 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.417511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-fa8af0-dockercfg-8bjb5\""
Sep 19 15:46:07.417671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.417630    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-fa8af0\""
Sep 19 15:46:07.428643 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.428624    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"]
Sep 19 15:46:07.452378 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.452357    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"]
Sep 19 15:46:07.455647 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.455632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:07.465516 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.465488    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"]
Sep 19 15:46:07.489697 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.489671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmx7\" (UniqueName: \"kubernetes.io/projected/24d57936-7c1c-4752-9505-05e53e505d44-kube-api-access-npmx7\") pod \"payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n\" (UID: \"24d57936-7c1c-4752-9505-05e53e505d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:07.489796 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.489725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d57936-7c1c-4752-9505-05e53e505d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n\" (UID: \"24d57936-7c1c-4752-9505-05e53e505d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:07.590318 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.590289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj\" (UID: \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:07.590417 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.590337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmx7\" (UniqueName: \"kubernetes.io/projected/24d57936-7c1c-4752-9505-05e53e505d44-kube-api-access-npmx7\") pod \"payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n\" (UID: \"24d57936-7c1c-4752-9505-05e53e505d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:07.590417 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.590374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d57936-7c1c-4752-9505-05e53e505d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n\" (UID: \"24d57936-7c1c-4752-9505-05e53e505d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:07.590506 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.590434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9c6\" (UniqueName: \"kubernetes.io/projected/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-kube-api-access-4m9c6\") pod \"payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj\" (UID: \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:07.591142 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.591125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d57936-7c1c-4752-9505-05e53e505d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n\" (UID: \"24d57936-7c1c-4752-9505-05e53e505d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:07.598160 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.598136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmx7\" (UniqueName: \"kubernetes.io/projected/24d57936-7c1c-4752-9505-05e53e505d44-kube-api-access-npmx7\") pod \"payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n\" (UID: \"24d57936-7c1c-4752-9505-05e53e505d44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:07.690814 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.690790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9c6\" (UniqueName: \"kubernetes.io/projected/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-kube-api-access-4m9c6\") pod \"payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj\" (UID: \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:07.690895 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.690830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj\" (UID: \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:07.691382 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.691366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj\" (UID: \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:07.697912 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.697893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9c6\" (UniqueName: \"kubernetes.io/projected/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-kube-api-access-4m9c6\") pod \"payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj\" (UID: \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:07.724698 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.724677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:07.768483 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.768454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:07.994507 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:07.994484    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"]
Sep 19 15:46:07.996999 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:46:07.996967    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d57936_7c1c_4752_9505_05e53e505d44.slice/crio-ac47b8549678d0c98e845ce42956c5ab0b2729e57800c198ce2c0121efb60012 WatchSource:0}: Error finding container ac47b8549678d0c98e845ce42956c5ab0b2729e57800c198ce2c0121efb60012: Status 404 returned error can't find the container with id ac47b8549678d0c98e845ce42956c5ab0b2729e57800c198ce2c0121efb60012
Sep 19 15:46:08.041731 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.041710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"]
Sep 19 15:46:08.042807 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:46:08.042782    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1190f1_4c88_4021_ac4a_2a8d21fd5947.slice/crio-785f9ba02401d62fd1d133ff73575e7e486391a17dae5adae1758b9dbfe91b5c WatchSource:0}: Error finding container 785f9ba02401d62fd1d133ff73575e7e486391a17dae5adae1758b9dbfe91b5c: Status 404 returned error can't find the container with id 785f9ba02401d62fd1d133ff73575e7e486391a17dae5adae1758b9dbfe91b5c
Sep 19 15:46:08.960551 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.960515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n" event={"ID":"24d57936-7c1c-4752-9505-05e53e505d44","Type":"ContainerStarted","Data":"ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d"}
Sep 19 15:46:08.961025 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.960626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n" event={"ID":"24d57936-7c1c-4752-9505-05e53e505d44","Type":"ContainerStarted","Data":"ac47b8549678d0c98e845ce42956c5ab0b2729e57800c198ce2c0121efb60012"}
Sep 19 15:46:08.961025 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.960664    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:08.962399 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.962372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj" event={"ID":"8d1190f1-4c88-4021-ac4a-2a8d21fd5947","Type":"ContainerStarted","Data":"0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46"}
Sep 19 15:46:08.962505 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.962423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj" event={"ID":"8d1190f1-4c88-4021-ac4a-2a8d21fd5947","Type":"ContainerStarted","Data":"785f9ba02401d62fd1d133ff73575e7e486391a17dae5adae1758b9dbfe91b5c"}
Sep 19 15:46:08.962505 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.962449    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:08.976457 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.976401    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n" podStartSLOduration=1.976388686 podStartE2EDuration="1.976388686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:46:08.974789581 +0000 UTC m=+2030.868353218" watchObservedRunningTime="2026-09-19 15:46:08.976388686 +0000 UTC m=+2030.869952326"
Sep 19 15:46:08.990908 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:08.990868    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj" podStartSLOduration=1.990854754 podStartE2EDuration="1.990854754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:46:08.98955016 +0000 UTC m=+2030.883113811" watchObservedRunningTime="2026-09-19 15:46:08.990854754 +0000 UTC m=+2030.884418393"
Sep 19 15:46:19.969247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:19.969221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:19.969674 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:19.969281    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:45.594870 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.594670    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"]
Sep 19 15:46:45.595422 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.594929    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj" podUID="8d1190f1-4c88-4021-ac4a-2a8d21fd5947" containerName="payload-processing" containerID="cri-o://0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46" gracePeriod=30
Sep 19 15:46:45.601537 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.601501    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"]
Sep 19 15:46:45.601938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.601895    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n" podUID="24d57936-7c1c-4752-9505-05e53e505d44" containerName="payload-pre-processing" containerID="cri-o://ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d" gracePeriod=30
Sep 19 15:46:45.871449 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.871428    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:45.874221 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.874204    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:45.890584 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.890181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m9c6\" (UniqueName: \"kubernetes.io/projected/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-kube-api-access-4m9c6\") pod \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\" (UID: \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\") "
Sep 19 15:46:45.890584 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.890241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-plugins-config-volume\") pod \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\" (UID: \"8d1190f1-4c88-4021-ac4a-2a8d21fd5947\") "
Sep 19 15:46:45.890758 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.890701    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d1190f1-4c88-4021-ac4a-2a8d21fd5947" (UID: "8d1190f1-4c88-4021-ac4a-2a8d21fd5947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:46:45.892518 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.892489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-kube-api-access-4m9c6" (OuterVolumeSpecName: "kube-api-access-4m9c6") pod "8d1190f1-4c88-4021-ac4a-2a8d21fd5947" (UID: "8d1190f1-4c88-4021-ac4a-2a8d21fd5947"). InnerVolumeSpecName "kube-api-access-4m9c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:46:45.991404 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.991381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d57936-7c1c-4752-9505-05e53e505d44-plugins-config-volume\") pod \"24d57936-7c1c-4752-9505-05e53e505d44\" (UID: \"24d57936-7c1c-4752-9505-05e53e505d44\") "
Sep 19 15:46:45.991512 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.991418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npmx7\" (UniqueName: \"kubernetes.io/projected/24d57936-7c1c-4752-9505-05e53e505d44-kube-api-access-npmx7\") pod \"24d57936-7c1c-4752-9505-05e53e505d44\" (UID: \"24d57936-7c1c-4752-9505-05e53e505d44\") "
Sep 19 15:46:45.991582 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.991572    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m9c6\" (UniqueName: \"kubernetes.io/projected/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-kube-api-access-4m9c6\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:46:45.991644 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.991584    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1190f1-4c88-4021-ac4a-2a8d21fd5947-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:46:45.991754 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.991731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d57936-7c1c-4752-9505-05e53e505d44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24d57936-7c1c-4752-9505-05e53e505d44" (UID: "24d57936-7c1c-4752-9505-05e53e505d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:46:45.993450 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:45.993432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d57936-7c1c-4752-9505-05e53e505d44-kube-api-access-npmx7" (OuterVolumeSpecName: "kube-api-access-npmx7") pod "24d57936-7c1c-4752-9505-05e53e505d44" (UID: "24d57936-7c1c-4752-9505-05e53e505d44"). InnerVolumeSpecName "kube-api-access-npmx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:46:46.092247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.092224    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24d57936-7c1c-4752-9505-05e53e505d44-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:46:46.092247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.092246    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npmx7\" (UniqueName: \"kubernetes.io/projected/24d57936-7c1c-4752-9505-05e53e505d44-kube-api-access-npmx7\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:46:46.135581 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.135528    2579 generic.go:356] "Generic (PLEG): container finished" podID="24d57936-7c1c-4752-9505-05e53e505d44" containerID="ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d" exitCode=0
Sep 19 15:46:46.135581 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.135569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"
Sep 19 15:46:46.135763 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.135637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n" event={"ID":"24d57936-7c1c-4752-9505-05e53e505d44","Type":"ContainerDied","Data":"ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d"}
Sep 19 15:46:46.135763 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.135688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n" event={"ID":"24d57936-7c1c-4752-9505-05e53e505d44","Type":"ContainerDied","Data":"ac47b8549678d0c98e845ce42956c5ab0b2729e57800c198ce2c0121efb60012"}
Sep 19 15:46:46.135763 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.135713    2579 scope.go:117] "RemoveContainer" containerID="ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d"
Sep 19 15:46:46.136927 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.136908    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d1190f1-4c88-4021-ac4a-2a8d21fd5947" containerID="0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46" exitCode=0
Sep 19 15:46:46.137019 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.136950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj" event={"ID":"8d1190f1-4c88-4021-ac4a-2a8d21fd5947","Type":"ContainerDied","Data":"0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46"}
Sep 19 15:46:46.137019 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.136967    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"
Sep 19 15:46:46.137019 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.136988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj" event={"ID":"8d1190f1-4c88-4021-ac4a-2a8d21fd5947","Type":"ContainerDied","Data":"785f9ba02401d62fd1d133ff73575e7e486391a17dae5adae1758b9dbfe91b5c"}
Sep 19 15:46:46.144313 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.144289    2579 scope.go:117] "RemoveContainer" containerID="ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d"
Sep 19 15:46:46.144736 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:46:46.144711    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d\": container with ID starting with ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d not found: ID does not exist" containerID="ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d"
Sep 19 15:46:46.144820 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.144743    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d"} err="failed to get container status \"ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d\": rpc error: code = NotFound desc = could not find container \"ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d\": container with ID starting with ec44a531b1a07d9c2f332d4123be6bea71146a0e5f06d120eb605525a274a53d not found: ID does not exist"
Sep 19 15:46:46.144820 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.144759    2579 scope.go:117] "RemoveContainer" containerID="0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46"
Sep 19 15:46:46.155014 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.154988    2579 scope.go:117] "RemoveContainer" containerID="0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46"
Sep 19 15:46:46.155304 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:46:46.155286    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46\": container with ID starting with 0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46 not found: ID does not exist" containerID="0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46"
Sep 19 15:46:46.155365 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.155310    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46"} err="failed to get container status \"0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46\": rpc error: code = NotFound desc = could not find container \"0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46\": container with ID starting with 0a7c2b8252786a5700fc6472483c7fa74cebb1f19ee39afc4e471aa04e697a46 not found: ID does not exist"
Sep 19 15:46:46.172042 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.172015    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"]
Sep 19 15:46:46.173360 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.173343    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-fa8af0-58775ccdd6-pj78n"]
Sep 19 15:46:46.185694 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.185669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"]
Sep 19 15:46:46.193432 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:46.193412    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-fa8af0-6974cb684c-mrhcj"]
Sep 19 15:46:48.170113 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:48.170084    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d57936-7c1c-4752-9505-05e53e505d44" path="/var/lib/kubelet/pods/24d57936-7c1c-4752-9505-05e53e505d44/volumes"
Sep 19 15:46:48.170492 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:46:48.170424    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1190f1-4c88-4021-ac4a-2a8d21fd5947" path="/var/lib/kubelet/pods/8d1190f1-4c88-4021-ac4a-2a8d21fd5947/volumes"
Sep 19 15:47:11.207003 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.206970    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"]
Sep 19 15:47:11.207382 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.207173    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67" podUID="8a8874df-a4c6-448b-a9ea-41d4c863facc" containerName="maas-api" containerID="cri-o://88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285" gracePeriod=30
Sep 19 15:47:11.255324 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.255295    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"]
Sep 19 15:47:11.255583 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.255529    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv" podUID="8b1367ca-56c9-4259-bfb7-4653908738d9" containerName="payload-processing" containerID="cri-o://ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71" gracePeriod=30
Sep 19 15:47:11.262378 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.262353    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"]
Sep 19 15:47:11.262664 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.262577    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn" podUID="c5dee583-72f2-4408-906c-338543636c49" containerName="payload-pre-processing" containerID="cri-o://fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35" gracePeriod=30
Sep 19 15:47:11.533448 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.533423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:47:11.536278 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.536258    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:47:11.554977 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.554952    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:47:11.584173 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.584153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6t4x\" (UniqueName: \"kubernetes.io/projected/8b1367ca-56c9-4259-bfb7-4653908738d9-kube-api-access-b6t4x\") pod \"8b1367ca-56c9-4259-bfb7-4653908738d9\" (UID: \"8b1367ca-56c9-4259-bfb7-4653908738d9\") "
Sep 19 15:47:11.584283 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.584191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1367ca-56c9-4259-bfb7-4653908738d9-plugins-config-volume\") pod \"8b1367ca-56c9-4259-bfb7-4653908738d9\" (UID: \"8b1367ca-56c9-4259-bfb7-4653908738d9\") "
Sep 19 15:47:11.584684 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.584657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1367ca-56c9-4259-bfb7-4653908738d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b1367ca-56c9-4259-bfb7-4653908738d9" (UID: "8b1367ca-56c9-4259-bfb7-4653908738d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:47:11.586558 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.586532    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1367ca-56c9-4259-bfb7-4653908738d9-kube-api-access-b6t4x" (OuterVolumeSpecName: "kube-api-access-b6t4x") pod "8b1367ca-56c9-4259-bfb7-4653908738d9" (UID: "8b1367ca-56c9-4259-bfb7-4653908738d9"). InnerVolumeSpecName "kube-api-access-b6t4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:47:11.685592 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.685547    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-metrics-certs\") pod \"8a8874df-a4c6-448b-a9ea-41d4c863facc\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") "
Sep 19 15:47:11.685785 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.685642    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8874df-a4c6-448b-a9ea-41d4c863facc-openshift-service-ca\") pod \"8a8874df-a4c6-448b-a9ea-41d4c863facc\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") "
Sep 19 15:47:11.685785 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.685694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-maas-api-tls\") pod \"8a8874df-a4c6-448b-a9ea-41d4c863facc\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") "
Sep 19 15:47:11.685785 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.685722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlgld\" (UniqueName: \"kubernetes.io/projected/8a8874df-a4c6-448b-a9ea-41d4c863facc-kube-api-access-mlgld\") pod \"8a8874df-a4c6-448b-a9ea-41d4c863facc\" (UID: \"8a8874df-a4c6-448b-a9ea-41d4c863facc\") "
Sep 19 15:47:11.685785 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.685748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv5mv\" (UniqueName: \"kubernetes.io/projected/c5dee583-72f2-4408-906c-338543636c49-kube-api-access-mv5mv\") pod \"c5dee583-72f2-4408-906c-338543636c49\" (UID: \"c5dee583-72f2-4408-906c-338543636c49\") "
Sep 19 15:47:11.685989 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.685796    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dee583-72f2-4408-906c-338543636c49-plugins-config-volume\") pod \"c5dee583-72f2-4408-906c-338543636c49\" (UID: \"c5dee583-72f2-4408-906c-338543636c49\") "
Sep 19 15:47:11.686080 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.686044    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8874df-a4c6-448b-a9ea-41d4c863facc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a8874df-a4c6-448b-a9ea-41d4c863facc" (UID: "8a8874df-a4c6-448b-a9ea-41d4c863facc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:47:11.686186 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.686067    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6t4x\" (UniqueName: \"kubernetes.io/projected/8b1367ca-56c9-4259-bfb7-4653908738d9-kube-api-access-b6t4x\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:47:11.686496 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.686404    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5dee583-72f2-4408-906c-338543636c49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5dee583-72f2-4408-906c-338543636c49" (UID: "c5dee583-72f2-4408-906c-338543636c49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:47:11.686496 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.686509    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1367ca-56c9-4259-bfb7-4653908738d9-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:47:11.687924 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.687897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a8874df-a4c6-448b-a9ea-41d4c863facc" (UID: "8a8874df-a4c6-448b-a9ea-41d4c863facc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:47:11.688005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.687941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a8874df-a4c6-448b-a9ea-41d4c863facc" (UID: "8a8874df-a4c6-448b-a9ea-41d4c863facc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:47:11.688005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.687947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5dee583-72f2-4408-906c-338543636c49-kube-api-access-mv5mv" (OuterVolumeSpecName: "kube-api-access-mv5mv") pod "c5dee583-72f2-4408-906c-338543636c49" (UID: "c5dee583-72f2-4408-906c-338543636c49"). InnerVolumeSpecName "kube-api-access-mv5mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:47:11.688005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.687952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8874df-a4c6-448b-a9ea-41d4c863facc-kube-api-access-mlgld" (OuterVolumeSpecName: "kube-api-access-mlgld") pod "8a8874df-a4c6-448b-a9ea-41d4c863facc" (UID: "8a8874df-a4c6-448b-a9ea-41d4c863facc"). InnerVolumeSpecName "kube-api-access-mlgld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:47:11.787569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.787497    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-maas-api-tls\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:47:11.787569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.787534    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlgld\" (UniqueName: \"kubernetes.io/projected/8a8874df-a4c6-448b-a9ea-41d4c863facc-kube-api-access-mlgld\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:47:11.787569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.787544    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv5mv\" (UniqueName: \"kubernetes.io/projected/c5dee583-72f2-4408-906c-338543636c49-kube-api-access-mv5mv\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:47:11.787569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.787552    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dee583-72f2-4408-906c-338543636c49-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:47:11.787569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.787560    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8874df-a4c6-448b-a9ea-41d4c863facc-metrics-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:47:11.787569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:11.787568    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8874df-a4c6-448b-a9ea-41d4c863facc-openshift-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:47:12.265344 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.265313    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5dee583-72f2-4408-906c-338543636c49" containerID="fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35" exitCode=0
Sep 19 15:47:12.265738 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.265360    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"
Sep 19 15:47:12.265738 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.265407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn" event={"ID":"c5dee583-72f2-4408-906c-338543636c49","Type":"ContainerDied","Data":"fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35"}
Sep 19 15:47:12.265738 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.265472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn" event={"ID":"c5dee583-72f2-4408-906c-338543636c49","Type":"ContainerDied","Data":"54a362ad911ed8ca68079ec525c3df10497bbdee57faa6193befa8636417fb38"}
Sep 19 15:47:12.265738 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.265506    2579 scope.go:117] "RemoveContainer" containerID="fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35"
Sep 19 15:47:12.266987 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.266893    2579 generic.go:356] "Generic (PLEG): container finished" podID="8b1367ca-56c9-4259-bfb7-4653908738d9" containerID="ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71" exitCode=0
Sep 19 15:47:12.266987 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.266928    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"
Sep 19 15:47:12.266987 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.266973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv" event={"ID":"8b1367ca-56c9-4259-bfb7-4653908738d9","Type":"ContainerDied","Data":"ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71"}
Sep 19 15:47:12.267171 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.267016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv" event={"ID":"8b1367ca-56c9-4259-bfb7-4653908738d9","Type":"ContainerDied","Data":"373a9321337c3be6a0515e1a3c202e3b8e237374fc33cc089ea0106f8b342b44"}
Sep 19 15:47:12.268521 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.268413    2579 generic.go:356] "Generic (PLEG): container finished" podID="8a8874df-a4c6-448b-a9ea-41d4c863facc" containerID="88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285" exitCode=0
Sep 19 15:47:12.268521 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.268455    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"
Sep 19 15:47:12.268521 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.268459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67" event={"ID":"8a8874df-a4c6-448b-a9ea-41d4c863facc","Type":"ContainerDied","Data":"88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285"}
Sep 19 15:47:12.268521 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.268498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67" event={"ID":"8a8874df-a4c6-448b-a9ea-41d4c863facc","Type":"ContainerDied","Data":"506af70200d2bf87375d9fc4b7ebe2396e6cab57c3e1cac033fe8fe9d825c6d6"}
Sep 19 15:47:12.275584 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.275567    2579 scope.go:117] "RemoveContainer" containerID="fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35"
Sep 19 15:47:12.275869 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:47:12.275850    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35\": container with ID starting with fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35 not found: ID does not exist" containerID="fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35"
Sep 19 15:47:12.275931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.275876    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35"} err="failed to get container status \"fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35\": rpc error: code = NotFound desc = could not find container \"fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35\": container with ID starting with fb200b77ee87336ef5274adfd1bc2b226775c343bcb0a62947d4d76dfbf64e35 not found: ID does not exist"
Sep 19 15:47:12.275931 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.275898    2579 scope.go:117] "RemoveContainer" containerID="ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71"
Sep 19 15:47:12.283741 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.283712    2579 scope.go:117] "RemoveContainer" containerID="ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71"
Sep 19 15:47:12.284027 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:47:12.284005    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71\": container with ID starting with ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71 not found: ID does not exist" containerID="ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71"
Sep 19 15:47:12.284098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.284037    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71"} err="failed to get container status \"ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71\": rpc error: code = NotFound desc = could not find container \"ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71\": container with ID starting with ca04b514ab3940d543ca3278b6ad244e3c918382cd4d6136c799e7d8fd8adb71 not found: ID does not exist"
Sep 19 15:47:12.284098 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.284058    2579 scope.go:117] "RemoveContainer" containerID="88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285"
Sep 19 15:47:12.285259 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.285241    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"]
Sep 19 15:47:12.292195 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.292173    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-521326-7655f5868b-js2wn"]
Sep 19 15:47:12.292827 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.292805    2579 scope.go:117] "RemoveContainer" containerID="88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285"
Sep 19 15:47:12.293092 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:47:12.293073    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285\": container with ID starting with 88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285 not found: ID does not exist" containerID="88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285"
Sep 19 15:47:12.293159 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.293102    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285"} err="failed to get container status \"88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285\": rpc error: code = NotFound desc = could not find container \"88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285\": container with ID starting with 88b76e05dbda7269e865685f5c7933e094ccf4d1d3a1a3c788c15d947a088285 not found: ID does not exist"
Sep 19 15:47:12.301678 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.301657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"]
Sep 19 15:47:12.306096 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.306073    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-521326-77c7dc5888-qzt67"]
Sep 19 15:47:12.314787 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.314768    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"]
Sep 19 15:47:12.318934 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:12.318916    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-521326-79cf58d6c-9pkvv"]
Sep 19 15:47:14.174997 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:14.174954    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a8874df-a4c6-448b-a9ea-41d4c863facc" path="/var/lib/kubelet/pods/8a8874df-a4c6-448b-a9ea-41d4c863facc/volumes"
Sep 19 15:47:14.175479 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:14.175436    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b1367ca-56c9-4259-bfb7-4653908738d9" path="/var/lib/kubelet/pods/8b1367ca-56c9-4259-bfb7-4653908738d9/volumes"
Sep 19 15:47:14.175846 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:14.175822    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5dee583-72f2-4408-906c-338543636c49" path="/var/lib/kubelet/pods/c5dee583-72f2-4408-906c-338543636c49/volumes"
Sep 19 15:47:20.252543 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:20.252516    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:47:20.257340 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:47:20.257320    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:48:10.772398 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:10.772369    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"]
Sep 19 15:48:10.772838 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:10.772631    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh" podUID="a8cad06f-a1a9-4021-a803-5757185d6d06" containerName="maas-api" containerID="cri-o://62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1" gracePeriod=30
Sep 19 15:48:10.817623 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:10.817572    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"]
Sep 19 15:48:10.817828 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:10.817790    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6" podUID="f39b3662-9bfb-437e-833f-d9d1e2097eda" containerName="payload-processing" containerID="cri-o://83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013" gracePeriod=30
Sep 19 15:48:10.823461 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:10.823435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"]
Sep 19 15:48:10.823704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:10.823677    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw" podUID="e4768cb4-1f3f-49d5-9f51-cfdefee28ae6" containerName="payload-pre-processing" containerID="cri-o://c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387" gracePeriod=30
Sep 19 15:48:11.075817 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.075785    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:48:11.095794 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.095767    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:48:11.102028 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.102003    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:48:11.120174 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120145    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qqrq\" (UniqueName: \"kubernetes.io/projected/a8cad06f-a1a9-4021-a803-5757185d6d06-kube-api-access-7qqrq\") pod \"a8cad06f-a1a9-4021-a803-5757185d6d06\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") "
Sep 19 15:48:11.120265 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbz5l\" (UniqueName: \"kubernetes.io/projected/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-kube-api-access-rbz5l\") pod \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\" (UID: \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\") "
Sep 19 15:48:11.120305 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39b3662-9bfb-437e-833f-d9d1e2097eda-plugins-config-volume\") pod \"f39b3662-9bfb-437e-833f-d9d1e2097eda\" (UID: \"f39b3662-9bfb-437e-833f-d9d1e2097eda\") "
Sep 19 15:48:11.120346 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cad06f-a1a9-4021-a803-5757185d6d06-openshift-service-ca\") pod \"a8cad06f-a1a9-4021-a803-5757185d6d06\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") "
Sep 19 15:48:11.120382 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120353    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls\") pod \"a8cad06f-a1a9-4021-a803-5757185d6d06\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") "
Sep 19 15:48:11.120434 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120380    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-plugins-config-volume\") pod \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\" (UID: \"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6\") "
Sep 19 15:48:11.120434 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120413    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-metrics-certs\") pod \"a8cad06f-a1a9-4021-a803-5757185d6d06\" (UID: \"a8cad06f-a1a9-4021-a803-5757185d6d06\") "
Sep 19 15:48:11.120544 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpkvb\" (UniqueName: \"kubernetes.io/projected/f39b3662-9bfb-437e-833f-d9d1e2097eda-kube-api-access-tpkvb\") pod \"f39b3662-9bfb-437e-833f-d9d1e2097eda\" (UID: \"f39b3662-9bfb-437e-833f-d9d1e2097eda\") "
Sep 19 15:48:11.120787 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120749    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cad06f-a1a9-4021-a803-5757185d6d06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8cad06f-a1a9-4021-a803-5757185d6d06" (UID: "a8cad06f-a1a9-4021-a803-5757185d6d06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:48:11.120870 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39b3662-9bfb-437e-833f-d9d1e2097eda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f39b3662-9bfb-437e-833f-d9d1e2097eda" (UID: "f39b3662-9bfb-437e-833f-d9d1e2097eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:48:11.120870 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.120843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4768cb4-1f3f-49d5-9f51-cfdefee28ae6" (UID: "e4768cb4-1f3f-49d5-9f51-cfdefee28ae6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:48:11.123495 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.123473    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cad06f-a1a9-4021-a803-5757185d6d06-kube-api-access-7qqrq" (OuterVolumeSpecName: "kube-api-access-7qqrq") pod "a8cad06f-a1a9-4021-a803-5757185d6d06" (UID: "a8cad06f-a1a9-4021-a803-5757185d6d06"). InnerVolumeSpecName "kube-api-access-7qqrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:48:11.123711 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.123683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a8cad06f-a1a9-4021-a803-5757185d6d06" (UID: "a8cad06f-a1a9-4021-a803-5757185d6d06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:48:11.124036 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.123997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8cad06f-a1a9-4021-a803-5757185d6d06" (UID: "a8cad06f-a1a9-4021-a803-5757185d6d06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:48:11.124148 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.124082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39b3662-9bfb-437e-833f-d9d1e2097eda-kube-api-access-tpkvb" (OuterVolumeSpecName: "kube-api-access-tpkvb") pod "f39b3662-9bfb-437e-833f-d9d1e2097eda" (UID: "f39b3662-9bfb-437e-833f-d9d1e2097eda"). InnerVolumeSpecName "kube-api-access-tpkvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:48:11.124592 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.124245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-kube-api-access-rbz5l" (OuterVolumeSpecName: "kube-api-access-rbz5l") pod "e4768cb4-1f3f-49d5-9f51-cfdefee28ae6" (UID: "e4768cb4-1f3f-49d5-9f51-cfdefee28ae6"). InnerVolumeSpecName "kube-api-access-rbz5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:48:11.221366 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.221344    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-maas-api-tls\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:11.221366 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.221365    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:11.221489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.221374    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cad06f-a1a9-4021-a803-5757185d6d06-metrics-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:11.221489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.221383    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpkvb\" (UniqueName: \"kubernetes.io/projected/f39b3662-9bfb-437e-833f-d9d1e2097eda-kube-api-access-tpkvb\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:11.221489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.221391    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qqrq\" (UniqueName: \"kubernetes.io/projected/a8cad06f-a1a9-4021-a803-5757185d6d06-kube-api-access-7qqrq\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:11.221489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.221400    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbz5l\" (UniqueName: \"kubernetes.io/projected/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6-kube-api-access-rbz5l\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:11.221489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.221408    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39b3662-9bfb-437e-833f-d9d1e2097eda-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:11.221489 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.221417    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cad06f-a1a9-4021-a803-5757185d6d06-openshift-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:11.530867 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.530841    2579 generic.go:356] "Generic (PLEG): container finished" podID="f39b3662-9bfb-437e-833f-d9d1e2097eda" containerID="83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013" exitCode=0
Sep 19 15:48:11.530982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.530879    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"
Sep 19 15:48:11.530982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.530910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6" event={"ID":"f39b3662-9bfb-437e-833f-d9d1e2097eda","Type":"ContainerDied","Data":"83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013"}
Sep 19 15:48:11.530982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.530959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6" event={"ID":"f39b3662-9bfb-437e-833f-d9d1e2097eda","Type":"ContainerDied","Data":"63dc0f8b02d68b96fe932c401b189971b4af1f28098ea5cc33e3b9ab7e4161d5"}
Sep 19 15:48:11.531111 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.530989    2579 scope.go:117] "RemoveContainer" containerID="83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013"
Sep 19 15:48:11.532317 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.532292    2579 generic.go:356] "Generic (PLEG): container finished" podID="e4768cb4-1f3f-49d5-9f51-cfdefee28ae6" containerID="c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387" exitCode=0
Sep 19 15:48:11.532382 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.532336    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"
Sep 19 15:48:11.532434 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.532379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw" event={"ID":"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6","Type":"ContainerDied","Data":"c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387"}
Sep 19 15:48:11.532491 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.532439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw" event={"ID":"e4768cb4-1f3f-49d5-9f51-cfdefee28ae6","Type":"ContainerDied","Data":"1802b6042470bc06fb54b8aa27a2604729ca3f4d6d928ccafbed0fdc7de835c8"}
Sep 19 15:48:11.533890 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.533868    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8cad06f-a1a9-4021-a803-5757185d6d06" containerID="62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1" exitCode=0
Sep 19 15:48:11.533995 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.533907    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"
Sep 19 15:48:11.534053 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.533905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh" event={"ID":"a8cad06f-a1a9-4021-a803-5757185d6d06","Type":"ContainerDied","Data":"62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1"}
Sep 19 15:48:11.534104 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.534057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh" event={"ID":"a8cad06f-a1a9-4021-a803-5757185d6d06","Type":"ContainerDied","Data":"a14e3937d1748f59ae288780fab51f653f5dea235b909eccf8cf3b17de206bc2"}
Sep 19 15:48:11.544710 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.544686    2579 scope.go:117] "RemoveContainer" containerID="83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013"
Sep 19 15:48:11.545033 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:48:11.545009    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013\": container with ID starting with 83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013 not found: ID does not exist" containerID="83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013"
Sep 19 15:48:11.545154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.545043    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013"} err="failed to get container status \"83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013\": rpc error: code = NotFound desc = could not find container \"83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013\": container with ID starting with 83559881bbb0ae48fd55b000d8f43ce3e8a82c58782b6e64c0c93033278de013 not found: ID does not exist"
Sep 19 15:48:11.545154 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.545068    2579 scope.go:117] "RemoveContainer" containerID="c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387"
Sep 19 15:48:11.552991 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.552973    2579 scope.go:117] "RemoveContainer" containerID="c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387"
Sep 19 15:48:11.553232 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:48:11.553216    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387\": container with ID starting with c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387 not found: ID does not exist" containerID="c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387"
Sep 19 15:48:11.553280 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.553239    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387"} err="failed to get container status \"c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387\": rpc error: code = NotFound desc = could not find container \"c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387\": container with ID starting with c2f528eade75a66ca72af3eb2dfc3d30237de6dae9a2b6d96abe8932bb742387 not found: ID does not exist"
Sep 19 15:48:11.553280 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.553254    2579 scope.go:117] "RemoveContainer" containerID="62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1"
Sep 19 15:48:11.556778 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.556754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"]
Sep 19 15:48:11.560854 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.560837    2579 scope.go:117] "RemoveContainer" containerID="62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1"
Sep 19 15:48:11.561198 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:48:11.561171    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1\": container with ID starting with 62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1 not found: ID does not exist" containerID="62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1"
Sep 19 15:48:11.561284 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.561208    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1"} err="failed to get container status \"62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1\": rpc error: code = NotFound desc = could not find container \"62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1\": container with ID starting with 62219fef6eb82c7695791898a40f2424fb340d5fa024a2d9263a3c928f51fcb1 not found: ID does not exist"
Sep 19 15:48:11.563087 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.563066    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-30ef69-778f7878d6-g9xnw"]
Sep 19 15:48:11.571977 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.571958    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"]
Sep 19 15:48:11.575479 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.575459    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30ef69-78b476ddb5-7qwr6"]
Sep 19 15:48:11.584042 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.584021    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"]
Sep 19 15:48:11.588164 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:11.588142    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30ef69-64b6b6475f-w66sh"]
Sep 19 15:48:12.171230 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:12.171156    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cad06f-a1a9-4021-a803-5757185d6d06" path="/var/lib/kubelet/pods/a8cad06f-a1a9-4021-a803-5757185d6d06/volumes"
Sep 19 15:48:12.171535 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:12.171501    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4768cb4-1f3f-49d5-9f51-cfdefee28ae6" path="/var/lib/kubelet/pods/e4768cb4-1f3f-49d5-9f51-cfdefee28ae6/volumes"
Sep 19 15:48:12.171822 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:12.171810    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39b3662-9bfb-437e-833f-d9d1e2097eda" path="/var/lib/kubelet/pods/f39b3662-9bfb-437e-833f-d9d1e2097eda/volumes"
Sep 19 15:48:28.245156 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:28.245121    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"]
Sep 19 15:48:28.248176 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:28.248153    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-50372a-29830545-cr2j8"]
Sep 19 15:48:30.170823 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:30.170793    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8af884a-4516-41ed-81c5-4056ca83452d" path="/var/lib/kubelet/pods/d8af884a-4516-41ed-81c5-4056ca83452d/volumes"
Sep 19 15:48:45.759965 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:45.759935    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"]
Sep 19 15:48:45.760468 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:45.760148    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp" podUID="0e6e577a-133c-4b7a-80c7-d2072c826c5e" containerName="payload-processing" containerID="cri-o://87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e" gracePeriod=30
Sep 19 15:48:45.767595 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:45.767564    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"]
Sep 19 15:48:45.768114 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:45.767793    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp" podUID="08a5468b-22ad-4499-b097-cce315e3a397" containerName="payload-pre-processing" containerID="cri-o://f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f" gracePeriod=30
Sep 19 15:48:46.033524 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.033503    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:48:46.036196 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.036176    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:48:46.178932 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.178904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a5468b-22ad-4499-b097-cce315e3a397-plugins-config-volume\") pod \"08a5468b-22ad-4499-b097-cce315e3a397\" (UID: \"08a5468b-22ad-4499-b097-cce315e3a397\") "
Sep 19 15:48:46.179066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.178945    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6e577a-133c-4b7a-80c7-d2072c826c5e-plugins-config-volume\") pod \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\" (UID: \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\") "
Sep 19 15:48:46.179066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.179015    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w55bs\" (UniqueName: \"kubernetes.io/projected/0e6e577a-133c-4b7a-80c7-d2072c826c5e-kube-api-access-w55bs\") pod \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\" (UID: \"0e6e577a-133c-4b7a-80c7-d2072c826c5e\") "
Sep 19 15:48:46.179066 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.179038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gqm6\" (UniqueName: \"kubernetes.io/projected/08a5468b-22ad-4499-b097-cce315e3a397-kube-api-access-6gqm6\") pod \"08a5468b-22ad-4499-b097-cce315e3a397\" (UID: \"08a5468b-22ad-4499-b097-cce315e3a397\") "
Sep 19 15:48:46.179331 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.179298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a5468b-22ad-4499-b097-cce315e3a397-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08a5468b-22ad-4499-b097-cce315e3a397" (UID: "08a5468b-22ad-4499-b097-cce315e3a397"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:48:46.179448 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.179344    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6e577a-133c-4b7a-80c7-d2072c826c5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e6e577a-133c-4b7a-80c7-d2072c826c5e" (UID: "0e6e577a-133c-4b7a-80c7-d2072c826c5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:48:46.181164 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.181144    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6e577a-133c-4b7a-80c7-d2072c826c5e-kube-api-access-w55bs" (OuterVolumeSpecName: "kube-api-access-w55bs") pod "0e6e577a-133c-4b7a-80c7-d2072c826c5e" (UID: "0e6e577a-133c-4b7a-80c7-d2072c826c5e"). InnerVolumeSpecName "kube-api-access-w55bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:48:46.181263 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.181242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a5468b-22ad-4499-b097-cce315e3a397-kube-api-access-6gqm6" (OuterVolumeSpecName: "kube-api-access-6gqm6") pod "08a5468b-22ad-4499-b097-cce315e3a397" (UID: "08a5468b-22ad-4499-b097-cce315e3a397"). InnerVolumeSpecName "kube-api-access-6gqm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:48:46.280341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.280283    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55bs\" (UniqueName: \"kubernetes.io/projected/0e6e577a-133c-4b7a-80c7-d2072c826c5e-kube-api-access-w55bs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:46.280341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.280307    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqm6\" (UniqueName: \"kubernetes.io/projected/08a5468b-22ad-4499-b097-cce315e3a397-kube-api-access-6gqm6\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:46.280341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.280323    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a5468b-22ad-4499-b097-cce315e3a397-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:46.280341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.280335    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6e577a-133c-4b7a-80c7-d2072c826c5e-plugins-config-volume\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:48:46.684076 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.684038    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e6e577a-133c-4b7a-80c7-d2072c826c5e" containerID="87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e" exitCode=0
Sep 19 15:48:46.684246 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.684091    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"
Sep 19 15:48:46.684246 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.684118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp" event={"ID":"0e6e577a-133c-4b7a-80c7-d2072c826c5e","Type":"ContainerDied","Data":"87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e"}
Sep 19 15:48:46.684246 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.684187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp" event={"ID":"0e6e577a-133c-4b7a-80c7-d2072c826c5e","Type":"ContainerDied","Data":"5a7cec78526c205552c9f16429c8ab2f5e1e7b90982c6206737603f7df1ca0f3"}
Sep 19 15:48:46.684246 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.684209    2579 scope.go:117] "RemoveContainer" containerID="87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e"
Sep 19 15:48:46.685585 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.685560    2579 generic.go:356] "Generic (PLEG): container finished" podID="08a5468b-22ad-4499-b097-cce315e3a397" containerID="f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f" exitCode=0
Sep 19 15:48:46.685711 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.685595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp" event={"ID":"08a5468b-22ad-4499-b097-cce315e3a397","Type":"ContainerDied","Data":"f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f"}
Sep 19 15:48:46.685711 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.685648    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"
Sep 19 15:48:46.685711 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.685675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp" event={"ID":"08a5468b-22ad-4499-b097-cce315e3a397","Type":"ContainerDied","Data":"f0eac800460bf350bbf76a643a33ad0cae8cb67c20436764a126e7dcb4034850"}
Sep 19 15:48:46.692723 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.692707    2579 scope.go:117] "RemoveContainer" containerID="87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e"
Sep 19 15:48:46.693013 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:48:46.692994    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e\": container with ID starting with 87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e not found: ID does not exist" containerID="87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e"
Sep 19 15:48:46.693068 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.693022    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e"} err="failed to get container status \"87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e\": rpc error: code = NotFound desc = could not find container \"87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e\": container with ID starting with 87787231a5f1b11de8ebc88cc623badc2cef40255a71963112ce28d01905e33e not found: ID does not exist"
Sep 19 15:48:46.693068 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.693037    2579 scope.go:117] "RemoveContainer" containerID="f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f"
Sep 19 15:48:46.700979 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.700964    2579 scope.go:117] "RemoveContainer" containerID="f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f"
Sep 19 15:48:46.701232 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:48:46.701215    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f\": container with ID starting with f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f not found: ID does not exist" containerID="f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f"
Sep 19 15:48:46.701281 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.701241    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f"} err="failed to get container status \"f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f\": rpc error: code = NotFound desc = could not find container \"f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f\": container with ID starting with f0fb149a3ea52bc6094ae4b5a49c38772b2dab77520b7099f01e1b8e17eab20f not found: ID does not exist"
Sep 19 15:48:46.708006 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.707985    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"]
Sep 19 15:48:46.711824 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.711804    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9c5c2-95f4d4d6c-vgqsp"]
Sep 19 15:48:46.720015 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.719995    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"]
Sep 19 15:48:46.724288 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:46.724267    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9c5c2-74bc4fc88b-vl7bp"]
Sep 19 15:48:48.170371 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:48.170343    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a5468b-22ad-4499-b097-cce315e3a397" path="/var/lib/kubelet/pods/08a5468b-22ad-4499-b097-cce315e3a397/volumes"
Sep 19 15:48:48.170724 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:48:48.170694    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6e577a-133c-4b7a-80c7-d2072c826c5e" path="/var/lib/kubelet/pods/0e6e577a-133c-4b7a-80c7-d2072c826c5e/volumes"
Sep 19 15:51:20.506741 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:51:20.506713    2579 scope.go:117] "RemoveContainer" containerID="b1a9c5d630d29053f2e0fe5afc3de7d718afccea5f30103d0d2c2d0a01b7659b"
Sep 19 15:52:20.279568 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:52:20.279541    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:52:20.284982 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:52:20.284959    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j5v77_d9d4c00c-9b8d-485f-836c-6af14fe6cd6e/ovn-acl-logging/0.log"
Sep 19 15:53:29.033877 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.033845    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"]
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034128    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d57936-7c1c-4752-9505-05e53e505d44" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d57936-7c1c-4752-9505-05e53e505d44" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034147    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1367ca-56c9-4259-bfb7-4653908738d9" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034152    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1367ca-56c9-4259-bfb7-4653908738d9" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034160    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6e577a-133c-4b7a-80c7-d2072c826c5e" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034165    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6e577a-133c-4b7a-80c7-d2072c826c5e" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034176    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a8874df-a4c6-448b-a9ea-41d4c863facc" containerName="maas-api"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034181    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a8874df-a4c6-448b-a9ea-41d4c863facc" containerName="maas-api"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034186    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cad06f-a1a9-4021-a803-5757185d6d06" containerName="maas-api"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034191    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cad06f-a1a9-4021-a803-5757185d6d06" containerName="maas-api"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034198    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4768cb4-1f3f-49d5-9f51-cfdefee28ae6" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034203    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4768cb4-1f3f-49d5-9f51-cfdefee28ae6" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034212    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a5468b-22ad-4499-b097-cce315e3a397" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034217    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a5468b-22ad-4499-b097-cce315e3a397" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034224    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5dee583-72f2-4408-906c-338543636c49" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034229    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5dee583-72f2-4408-906c-338543636c49" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034235    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39b3662-9bfb-437e-833f-d9d1e2097eda" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034239    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39b3662-9bfb-437e-833f-d9d1e2097eda" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034246    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1190f1-4c88-4021-ac4a-2a8d21fd5947" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034251    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1190f1-4c88-4021-ac4a-2a8d21fd5947" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034294    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6e577a-133c-4b7a-80c7-d2072c826c5e" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034302    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4768cb4-1f3f-49d5-9f51-cfdefee28ae6" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034308    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cad06f-a1a9-4021-a803-5757185d6d06" containerName="maas-api"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034314    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1190f1-4c88-4021-ac4a-2a8d21fd5947" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034320    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39b3662-9bfb-437e-833f-d9d1e2097eda" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034326    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1367ca-56c9-4259-bfb7-4653908738d9" containerName="payload-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034332    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a5468b-22ad-4499-b097-cce315e3a397" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034338    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a8874df-a4c6-448b-a9ea-41d4c863facc" containerName="maas-api"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034344    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5dee583-72f2-4408-906c-338543636c49" containerName="payload-pre-processing"
Sep 19 15:53:29.034387 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.034349    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d57936-7c1c-4752-9505-05e53e505d44" containerName="payload-pre-processing"
Sep 19 15:53:29.037316 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.037295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.039325 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.039300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-452b12f1-kserve-self-signed-certs\""
Sep 19 15:53:29.047072 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.047048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"]
Sep 19 15:53:29.168391 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.168369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmprp\" (UniqueName: \"kubernetes.io/projected/b04502fe-7066-47a9-a49b-9a17924da173-kube-api-access-rmprp\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.168509 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.168401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-tmp-dir\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.168509 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.168427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-home\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.168509 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.168441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-dshm\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.168509 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.168486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04502fe-7066-47a9-a49b-9a17924da173-tls-certs\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.168683 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.168517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-model-cache\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269102 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-home\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269172 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-dshm\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269172 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04502fe-7066-47a9-a49b-9a17924da173-tls-certs\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269172 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-model-cache\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269319 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmprp\" (UniqueName: \"kubernetes.io/projected/b04502fe-7066-47a9-a49b-9a17924da173-kube-api-access-rmprp\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269319 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-tmp-dir\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269530 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-home\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269597 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-tmp-dir\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.269681 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.269620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-model-cache\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.271487 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.271467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-dshm\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.271665 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.271650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04502fe-7066-47a9-a49b-9a17924da173-tls-certs\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.276704 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.276678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmprp\" (UniqueName: \"kubernetes.io/projected/b04502fe-7066-47a9-a49b-9a17924da173-kube-api-access-rmprp\") pod \"e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") " pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.346837 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.346789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:29.614415 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.614340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"]
Sep 19 15:53:29.616552 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:53:29.616519    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb04502fe_7066_47a9_a49b_9a17924da173.slice/crio-a9ecc2e6acec16495433cb35aca401e2cadee74c9c544cd96361f967c52b7259 WatchSource:0}: Error finding container a9ecc2e6acec16495433cb35aca401e2cadee74c9c544cd96361f967c52b7259: Status 404 returned error can't find the container with id a9ecc2e6acec16495433cb35aca401e2cadee74c9c544cd96361f967c52b7259
Sep 19 15:53:29.850342 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.850305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85" event={"ID":"b04502fe-7066-47a9-a49b-9a17924da173","Type":"ContainerStarted","Data":"9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f"}
Sep 19 15:53:29.850342 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.850348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85" event={"ID":"b04502fe-7066-47a9-a49b-9a17924da173","Type":"ContainerStarted","Data":"a9ecc2e6acec16495433cb35aca401e2cadee74c9c544cd96361f967c52b7259"}
Sep 19 15:53:29.865577 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:29.865492    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85" podStartSLOduration=0.865474082 podStartE2EDuration="865.474082ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:53:29.864626841 +0000 UTC m=+2471.758190475" watchObservedRunningTime="2026-09-19 15:53:29.865474082 +0000 UTC m=+2471.759037723"
Sep 19 15:53:39.347377 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:39.347336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:39.347879 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:39.347414    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:39.359358 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:39.359337    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:39.901796 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:39.901774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:53:55.461871 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:53:55.461836    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"]
Sep 19 15:54:10.480244 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.480187    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85" podUID="b04502fe-7066-47a9-a49b-9a17924da173" containerName="main" containerID="cri-o://9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f" gracePeriod=45
Sep 19 15:54:10.718365 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.718342    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:54:10.769537 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmprp\" (UniqueName: \"kubernetes.io/projected/b04502fe-7066-47a9-a49b-9a17924da173-kube-api-access-rmprp\") pod \"b04502fe-7066-47a9-a49b-9a17924da173\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") "
Sep 19 15:54:10.769537 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-tmp-dir\") pod \"b04502fe-7066-47a9-a49b-9a17924da173\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") "
Sep 19 15:54:10.769754 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-model-cache\") pod \"b04502fe-7066-47a9-a49b-9a17924da173\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") "
Sep 19 15:54:10.769754 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04502fe-7066-47a9-a49b-9a17924da173-tls-certs\") pod \"b04502fe-7066-47a9-a49b-9a17924da173\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") "
Sep 19 15:54:10.769754 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-dshm\") pod \"b04502fe-7066-47a9-a49b-9a17924da173\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") "
Sep 19 15:54:10.769754 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769684    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-home\") pod \"b04502fe-7066-47a9-a49b-9a17924da173\" (UID: \"b04502fe-7066-47a9-a49b-9a17924da173\") "
Sep 19 15:54:10.769940 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b04502fe-7066-47a9-a49b-9a17924da173" (UID: "b04502fe-7066-47a9-a49b-9a17924da173"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:54:10.769940 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-model-cache" (OuterVolumeSpecName: "model-cache") pod "b04502fe-7066-47a9-a49b-9a17924da173" (UID: "b04502fe-7066-47a9-a49b-9a17924da173"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:54:10.770030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769993    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-tmp-dir\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:10.770030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.769997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-home" (OuterVolumeSpecName: "home") pod "b04502fe-7066-47a9-a49b-9a17924da173" (UID: "b04502fe-7066-47a9-a49b-9a17924da173"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:54:10.770030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.770009    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-model-cache\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:10.771744 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.771725    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04502fe-7066-47a9-a49b-9a17924da173-kube-api-access-rmprp" (OuterVolumeSpecName: "kube-api-access-rmprp") pod "b04502fe-7066-47a9-a49b-9a17924da173" (UID: "b04502fe-7066-47a9-a49b-9a17924da173"). InnerVolumeSpecName "kube-api-access-rmprp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:54:10.771852 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.771835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04502fe-7066-47a9-a49b-9a17924da173-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b04502fe-7066-47a9-a49b-9a17924da173" (UID: "b04502fe-7066-47a9-a49b-9a17924da173"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:54:10.771921 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.771905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-dshm" (OuterVolumeSpecName: "dshm") pod "b04502fe-7066-47a9-a49b-9a17924da173" (UID: "b04502fe-7066-47a9-a49b-9a17924da173"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 15:54:10.870355 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.870332    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmprp\" (UniqueName: \"kubernetes.io/projected/b04502fe-7066-47a9-a49b-9a17924da173-kube-api-access-rmprp\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:10.870355 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.870353    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04502fe-7066-47a9-a49b-9a17924da173-tls-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:10.870482 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.870363    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-dshm\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:10.870482 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:10.870371    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04502fe-7066-47a9-a49b-9a17924da173-home\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:11.026165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.026110    2579 generic.go:356] "Generic (PLEG): container finished" podID="b04502fe-7066-47a9-a49b-9a17924da173" containerID="9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f" exitCode=0
Sep 19 15:54:11.026165 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.026150    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"
Sep 19 15:54:11.026297 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.026200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85" event={"ID":"b04502fe-7066-47a9-a49b-9a17924da173","Type":"ContainerDied","Data":"9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f"}
Sep 19 15:54:11.026297 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.026239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85" event={"ID":"b04502fe-7066-47a9-a49b-9a17924da173","Type":"ContainerDied","Data":"a9ecc2e6acec16495433cb35aca401e2cadee74c9c544cd96361f967c52b7259"}
Sep 19 15:54:11.026297 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.026255    2579 scope.go:117] "RemoveContainer" containerID="9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f"
Sep 19 15:54:11.034707 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.034691    2579 scope.go:117] "RemoveContainer" containerID="9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f"
Sep 19 15:54:11.034976 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:54:11.034959    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f\": container with ID starting with 9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f not found: ID does not exist" containerID="9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f"
Sep 19 15:54:11.035023 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.034988    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f"} err="failed to get container status \"9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f\": rpc error: code = NotFound desc = could not find container \"9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f\": container with ID starting with 9d8b4a18b9b4aa3abc4d4b52c28f1ad6a193d4b652cd1bf92380c9916a3b134f not found: ID does not exist"
Sep 19 15:54:11.045569 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.045550    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"]
Sep 19 15:54:11.049563 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:11.049542    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-452b12f1-kserve-fdbc999c5-8dj85"]
Sep 19 15:54:12.170049 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:12.170018    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b04502fe-7066-47a9-a49b-9a17924da173" path="/var/lib/kubelet/pods/b04502fe-7066-47a9-a49b-9a17924da173/volumes"
Sep 19 15:54:18.659769 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.659735    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"]
Sep 19 15:54:18.660121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.660027    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04502fe-7066-47a9-a49b-9a17924da173" containerName="main"
Sep 19 15:54:18.660121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.660037    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04502fe-7066-47a9-a49b-9a17924da173" containerName="main"
Sep 19 15:54:18.660121 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.660087    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04502fe-7066-47a9-a49b-9a17924da173" containerName="main"
Sep 19 15:54:18.662895 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.662859    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.664978 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.664958    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 19 15:54:18.665745 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.665719    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 19 15:54:18.665846 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.665783    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zhfqt\""
Sep 19 15:54:18.665846 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.665817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 19 15:54:18.665938 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.665783    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 19 15:54:18.673459 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.673436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"]
Sep 19 15:54:18.725491 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.725464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-maas-api-tls\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.725598 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.725505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-metrics-certs\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.725598 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.725532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbsp\" (UniqueName: \"kubernetes.io/projected/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-kube-api-access-qnbsp\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.725598 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.725578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-openshift-service-ca\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.826753 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.826731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-maas-api-tls\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.826852 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.826768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-metrics-certs\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.826852 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.826788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbsp\" (UniqueName: \"kubernetes.io/projected/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-kube-api-access-qnbsp\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.826852 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.826820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-openshift-service-ca\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.827486 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.827469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-openshift-service-ca\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.829282 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.829265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-maas-api-tls\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.829337 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.829312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-metrics-certs\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.847539 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.847515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbsp\" (UniqueName: \"kubernetes.io/projected/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-kube-api-access-qnbsp\") pod \"maas-api-f47b4794c-xgvf5\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") " pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:18.972213 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:18.972189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:19.240338 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:19.240262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"]
Sep 19 15:54:19.242950 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:54:19.242925    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb0b84f_6ba3_4c23_8b06_c2a65f2f6910.slice/crio-921fc0fddccd86d6fbae862d3c063571841d86c006e1e2934a2ee841ffde3b0b WatchSource:0}: Error finding container 921fc0fddccd86d6fbae862d3c063571841d86c006e1e2934a2ee841ffde3b0b: Status 404 returned error can't find the container with id 921fc0fddccd86d6fbae862d3c063571841d86c006e1e2934a2ee841ffde3b0b
Sep 19 15:54:19.244224 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:19.244206    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 15:54:20.073199 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:20.073161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5" event={"ID":"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910","Type":"ContainerStarted","Data":"22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce"}
Sep 19 15:54:20.073558 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:20.073255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:20.073558 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:20.073275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5" event={"ID":"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910","Type":"ContainerStarted","Data":"921fc0fddccd86d6fbae862d3c063571841d86c006e1e2934a2ee841ffde3b0b"}
Sep 19 15:54:20.089785 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:20.089739    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5" podStartSLOduration=1.6645246409999999 podStartE2EDuration="2.089725071s" totalImagesPullingTime="425.20043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:54:20.088391764 +0000 UTC m=+2521.981955404" watchObservedRunningTime="2026-09-19 15:54:20.089725071 +0000 UTC m=+2521.983288710"
Sep 19 15:54:26.081975 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.081948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:26.692501 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.692464    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"]
Sep 19 15:54:26.697269 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.697251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.705148 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.705094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"]
Sep 19 15:54:26.791085 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.791059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrwn\" (UniqueName: \"kubernetes.io/projected/22d0739c-2d30-48c0-b26a-1f257edba88b-kube-api-access-znrwn\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.791247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.791092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d0739c-2d30-48c0-b26a-1f257edba88b-maas-api-tls\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.791247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.791125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d0739c-2d30-48c0-b26a-1f257edba88b-openshift-service-ca\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.791247 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.791232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d0739c-2d30-48c0-b26a-1f257edba88b-metrics-certs\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.892092 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.892064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrwn\" (UniqueName: \"kubernetes.io/projected/22d0739c-2d30-48c0-b26a-1f257edba88b-kube-api-access-znrwn\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.892200 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.892095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d0739c-2d30-48c0-b26a-1f257edba88b-maas-api-tls\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.892200 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.892124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d0739c-2d30-48c0-b26a-1f257edba88b-openshift-service-ca\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.892200 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.892165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d0739c-2d30-48c0-b26a-1f257edba88b-metrics-certs\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.892865 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.892805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d0739c-2d30-48c0-b26a-1f257edba88b-openshift-service-ca\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.894697 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.894678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d0739c-2d30-48c0-b26a-1f257edba88b-metrics-certs\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.894792 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.894719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d0739c-2d30-48c0-b26a-1f257edba88b-maas-api-tls\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:26.899233 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:26.899216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrwn\" (UniqueName: \"kubernetes.io/projected/22d0739c-2d30-48c0-b26a-1f257edba88b-kube-api-access-znrwn\") pod \"maas-api-546445c5bc-tjfxr\" (UID: \"22d0739c-2d30-48c0-b26a-1f257edba88b\") " pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:27.009665 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:27.009591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:27.277527 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:27.277503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"]
Sep 19 15:54:27.279426 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:54:27.279394    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d0739c_2d30_48c0_b26a_1f257edba88b.slice/crio-0800287d5411a9f3225d1f4b8505f18b202244d6e5a9a3da179bfe07e2e6fc02 WatchSource:0}: Error finding container 0800287d5411a9f3225d1f4b8505f18b202244d6e5a9a3da179bfe07e2e6fc02: Status 404 returned error can't find the container with id 0800287d5411a9f3225d1f4b8505f18b202244d6e5a9a3da179bfe07e2e6fc02
Sep 19 15:54:28.109393 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:28.109359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr" event={"ID":"22d0739c-2d30-48c0-b26a-1f257edba88b","Type":"ContainerStarted","Data":"a8b36334cc8f1b6f770bafb66ca1b1a0dd1837b9340477d998031d6254a4222f"}
Sep 19 15:54:28.109554 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:28.109417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr" event={"ID":"22d0739c-2d30-48c0-b26a-1f257edba88b","Type":"ContainerStarted","Data":"0800287d5411a9f3225d1f4b8505f18b202244d6e5a9a3da179bfe07e2e6fc02"}
Sep 19 15:54:28.109554 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:28.109446    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:28.127576 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:28.127529    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr" podStartSLOduration=1.810661399 podStartE2EDuration="2.127514531s" totalImagesPullingTime="316.853132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 15:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:54:28.126036398 +0000 UTC m=+2530.019600048" watchObservedRunningTime="2026-09-19 15:54:28.127514531 +0000 UTC m=+2530.021078171"
Sep 19 15:54:34.119030 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.119002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-546445c5bc-tjfxr"
Sep 19 15:54:34.161788 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.161758    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"]
Sep 19 15:54:34.161989 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.161966    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5" podUID="3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" containerName="maas-api" containerID="cri-o://22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce" gracePeriod=30
Sep 19 15:54:34.399513 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.399490    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:34.554970 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.554946    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnbsp\" (UniqueName: \"kubernetes.io/projected/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-kube-api-access-qnbsp\") pod \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") "
Sep 19 15:54:34.555095 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.554975    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-openshift-service-ca\") pod \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") "
Sep 19 15:54:34.555095 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.555000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-maas-api-tls\") pod \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") "
Sep 19 15:54:34.555095 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.555032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-metrics-certs\") pod \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\" (UID: \"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910\") "
Sep 19 15:54:34.555276 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.555262    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" (UID: "3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 15:54:34.557135 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.557102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" (UID: "3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:54:34.557135 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.557120    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-kube-api-access-qnbsp" (OuterVolumeSpecName: "kube-api-access-qnbsp") pod "3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" (UID: "3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910"). InnerVolumeSpecName "kube-api-access-qnbsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 15:54:34.557279 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.557143    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" (UID: "3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 15:54:34.656005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.655956    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-metrics-certs\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:34.656005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.655975    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnbsp\" (UniqueName: \"kubernetes.io/projected/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-kube-api-access-qnbsp\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:34.656005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.655984    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-openshift-service-ca\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:34.656005 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:34.655993    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910-maas-api-tls\") on node \"ip-10-0-141-0.ec2.internal\" DevicePath \"\""
Sep 19 15:54:35.140180 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.140150    2579 generic.go:356] "Generic (PLEG): container finished" podID="3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" containerID="22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce" exitCode=0
Sep 19 15:54:35.140498 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.140202    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"
Sep 19 15:54:35.140498 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.140212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5" event={"ID":"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910","Type":"ContainerDied","Data":"22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce"}
Sep 19 15:54:35.140498 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.140250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5" event={"ID":"3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910","Type":"ContainerDied","Data":"921fc0fddccd86d6fbae862d3c063571841d86c006e1e2934a2ee841ffde3b0b"}
Sep 19 15:54:35.140498 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.140269    2579 scope.go:117] "RemoveContainer" containerID="22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce"
Sep 19 15:54:35.148826 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.148810    2579 scope.go:117] "RemoveContainer" containerID="22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce"
Sep 19 15:54:35.149074 ip-10-0-141-0 kubenswrapper[2579]: E0919 15:54:35.149058    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce\": container with ID starting with 22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce not found: ID does not exist" containerID="22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce"
Sep 19 15:54:35.149119 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.149081    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce"} err="failed to get container status \"22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce\": rpc error: code = NotFound desc = could not find container \"22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce\": container with ID starting with 22f5fc7ed4e59ebcc51404feeadcfc34c6d738cef15ff7d8bb5e71058ad030ce not found: ID does not exist"
Sep 19 15:54:35.160354 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.160330    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"]
Sep 19 15:54:35.161776 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:35.161756    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f47b4794c-xgvf5"]
Sep 19 15:54:36.169649 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:36.169618    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" path="/var/lib/kubelet/pods/3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910/volumes"
Sep 19 15:54:48.362202 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:48.362168    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85db744fc5-vhd4x_42419a3e-7e1a-4f7e-a111-1b0093b24e66/authorino/0.log"
Sep 19 15:54:54.918049 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:54.918023    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d8c7f66c-hq2hz_2eb599ba-05b4-47de-a16a-9fcb1a3a2b37/copy-manifests/0.log"
Sep 19 15:54:54.944179 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:54.944153    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d8c7f66c-hq2hz_2eb599ba-05b4-47de-a16a-9fcb1a3a2b37/manager/0.log"
Sep 19 15:54:55.526825 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:55.526794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-546445c5bc-tjfxr_22d0739c-2d30-48c0-b26a-1f257edba88b/maas-api/0.log"
Sep 19 15:54:55.633830 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:55.633798    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29830530-jh86d_a5cc9e7c-8431-4de2-9ffa-848f8bad93ee/cleanup/0.log"
Sep 19 15:54:56.596316 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:56.596280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85db744fc5-vhd4x_42419a3e-7e1a-4f7e-a111-1b0093b24e66/authorino/0.log"
Sep 19 15:54:56.921800 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:56.921718    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-btgkd_02e89a48-96ff-4fde-8ff3-9d278994582e/kuadrant-console-plugin/0.log"
Sep 19 15:54:57.835694 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:57.835667    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc4565f8d-dnjjq_8393d6f3-9666-4eae-9964-af24387a39fb/kube-auth-proxy/0.log"
Sep 19 15:54:59.141446 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:59.141415    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm_68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7/main/0.log"
Sep 19 15:54:59.363333 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:54:59.363303    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6crrt_bb9f2be9-d264-4d21-8255-bbcdf96e9152/main/0.log"
Sep 19 15:55:00.462560 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:00.462525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85db744fc5-vhd4x_42419a3e-7e1a-4f7e-a111-1b0093b24e66/authorino/0.log"
Sep 19 15:55:06.931401 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:06.931369    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d8c7f66c-hq2hz_2eb599ba-05b4-47de-a16a-9fcb1a3a2b37/copy-manifests/0.log"
Sep 19 15:55:06.943806 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:06.943783    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d8c7f66c-hq2hz_2eb599ba-05b4-47de-a16a-9fcb1a3a2b37/manager/0.log"
Sep 19 15:55:07.523231 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:07.523200    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-546445c5bc-tjfxr_22d0739c-2d30-48c0-b26a-1f257edba88b/maas-api/0.log"
Sep 19 15:55:07.638189 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:07.638171    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29830530-jh86d_a5cc9e7c-8431-4de2-9ffa-848f8bad93ee/cleanup/0.log"
Sep 19 15:55:08.610837 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:08.610811    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85db744fc5-vhd4x_42419a3e-7e1a-4f7e-a111-1b0093b24e66/authorino/0.log"
Sep 19 15:55:08.950219 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:08.950200    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-btgkd_02e89a48-96ff-4fde-8ff3-9d278994582e/kuadrant-console-plugin/0.log"
Sep 19 15:55:09.896341 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:09.896317    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc4565f8d-dnjjq_8393d6f3-9666-4eae-9964-af24387a39fb/kube-auth-proxy/0.log"
Sep 19 15:55:11.218780 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:11.218752    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ngtm_68ebd6b7-fe5b-4868-9fb9-fd22d8b000a7/main/0.log"
Sep 19 15:55:11.431819 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:11.431799    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6crrt_bb9f2be9-d264-4d21-8255-bbcdf96e9152/main/0.log"
Sep 19 15:55:32.701175 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:32.701097    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zmfsl_d9e06119-2d4a-42ce-b018-9f947d18ad98/global-pull-secret-syncer/0.log"
Sep 19 15:55:32.798230 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:32.798204    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7l4fc_61a4cb3c-f020-4393-9222-a814cece3949/konnectivity-agent/0.log"
Sep 19 15:55:32.898721 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:32.898700    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-0.ec2.internal_90d1d9526d2ae5083b8d03f7ccdd81d9/haproxy/0.log"
Sep 19 15:55:37.153486 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:37.153457    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85db744fc5-vhd4x_42419a3e-7e1a-4f7e-a111-1b0093b24e66/authorino/0.log"
Sep 19 15:55:37.240424 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:37.240405    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-btgkd_02e89a48-96ff-4fde-8ff3-9d278994582e/kuadrant-console-plugin/0.log"
Sep 19 15:55:39.053276 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.053245    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da849908-016d-4220-b2f2-f1dc5f2dc36e/alertmanager/0.log"
Sep 19 15:55:39.074867 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.074837    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da849908-016d-4220-b2f2-f1dc5f2dc36e/config-reloader/0.log"
Sep 19 15:55:39.094323 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.094300    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da849908-016d-4220-b2f2-f1dc5f2dc36e/kube-rbac-proxy-web/0.log"
Sep 19 15:55:39.114004 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.113981    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da849908-016d-4220-b2f2-f1dc5f2dc36e/kube-rbac-proxy/0.log"
Sep 19 15:55:39.133048 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.133021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da849908-016d-4220-b2f2-f1dc5f2dc36e/kube-rbac-proxy-metric/0.log"
Sep 19 15:55:39.153567 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.153534    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da849908-016d-4220-b2f2-f1dc5f2dc36e/prom-label-proxy/0.log"
Sep 19 15:55:39.176136 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.176119    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da849908-016d-4220-b2f2-f1dc5f2dc36e/init-config-reloader/0.log"
Sep 19 15:55:39.243037 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.243014    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-cmvt4_84c09b31-d6a3-42a3-9de5-5184c4e3b893/kube-state-metrics/0.log"
Sep 19 15:55:39.258927 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.258910    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-cmvt4_84c09b31-d6a3-42a3-9de5-5184c4e3b893/kube-rbac-proxy-main/0.log"
Sep 19 15:55:39.276747 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.276729    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-cmvt4_84c09b31-d6a3-42a3-9de5-5184c4e3b893/kube-rbac-proxy-self/0.log"
Sep 19 15:55:39.520678 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.520659    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsxd4_b8321440-eed7-41cd-8dfa-68c507247399/node-exporter/0.log"
Sep 19 15:55:39.538487 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.538469    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsxd4_b8321440-eed7-41cd-8dfa-68c507247399/kube-rbac-proxy/0.log"
Sep 19 15:55:39.559126 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.559109    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsxd4_b8321440-eed7-41cd-8dfa-68c507247399/init-textfile/0.log"
Sep 19 15:55:39.914366 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.914349    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-784764ddc8-5m2rp_0a124547-3cfa-498b-8230-f3edd90c5751/telemeter-client/0.log"
Sep 19 15:55:39.937254 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.937235    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-784764ddc8-5m2rp_0a124547-3cfa-498b-8230-f3edd90c5751/reload/0.log"
Sep 19 15:55:39.970204 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:39.970183    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-784764ddc8-5m2rp_0a124547-3cfa-498b-8230-f3edd90c5751/kube-rbac-proxy/0.log"
Sep 19 15:55:42.834671 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.834637    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"]
Sep 19 15:55:42.835022 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.834938    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" containerName="maas-api"
Sep 19 15:55:42.835022 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.834948    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" containerName="maas-api"
Sep 19 15:55:42.835022 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.835001    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb0b84f-6ba3-4c23-8b06-c2a65f2f6910" containerName="maas-api"
Sep 19 15:55:42.839101 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.839086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:42.840890 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.840870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-79whr\"/\"kube-root-ca.crt\""
Sep 19 15:55:42.841502 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.841477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-79whr\"/\"default-dockercfg-dqvw7\""
Sep 19 15:55:42.841589 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.841536    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-79whr\"/\"openshift-service-ca.crt\""
Sep 19 15:55:42.848184 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.848158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"]
Sep 19 15:55:42.948364 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.948344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-lib-modules\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:42.948481 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.948371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9phr\" (UniqueName: \"kubernetes.io/projected/f1f857c1-8223-4695-9511-d557ee1e2134-kube-api-access-c9phr\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:42.948481 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.948390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-podres\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:42.948481 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.948406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-proc\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:42.948481 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:42.948431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-sys\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.048928 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.048897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-sys\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.049035 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.048965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-lib-modules\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.049035 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.048990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9phr\" (UniqueName: \"kubernetes.io/projected/f1f857c1-8223-4695-9511-d557ee1e2134-kube-api-access-c9phr\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.049035 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.049009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-podres\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.049035 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.049023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-proc\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.049035 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.049027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-sys\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.049266 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.049156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-podres\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.049266 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.049165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-proc\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.049266 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.049158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1f857c1-8223-4695-9511-d557ee1e2134-lib-modules\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.056850 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.056823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9phr\" (UniqueName: \"kubernetes.io/projected/f1f857c1-8223-4695-9511-d557ee1e2134-kube-api-access-c9phr\") pod \"perf-node-gather-daemonset-ldc2m\" (UID: \"f1f857c1-8223-4695-9511-d557ee1e2134\") " pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.150308 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.150244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:43.418084 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.418013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"]
Sep 19 15:55:43.421507 ip-10-0-141-0 kubenswrapper[2579]: W0919 15:55:43.421474    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1f857c1_8223_4695_9511_d557ee1e2134.slice/crio-260179c847fedbfce02598c1da555cb8a694ea47f94b5c580c4df193fcaaf419 WatchSource:0}: Error finding container 260179c847fedbfce02598c1da555cb8a694ea47f94b5c580c4df193fcaaf419: Status 404 returned error can't find the container with id 260179c847fedbfce02598c1da555cb8a694ea47f94b5c580c4df193fcaaf419
Sep 19 15:55:43.432107 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:43.432076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m" event={"ID":"f1f857c1-8223-4695-9511-d557ee1e2134","Type":"ContainerStarted","Data":"260179c847fedbfce02598c1da555cb8a694ea47f94b5c580c4df193fcaaf419"}
Sep 19 15:55:44.438146 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:44.438114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m" event={"ID":"f1f857c1-8223-4695-9511-d557ee1e2134","Type":"ContainerStarted","Data":"9a4c40f956a93ee456c14b93aef24e0fbeed37b85c7086dbdca6d79c8a7d2bfb"}
Sep 19 15:55:44.438534 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:44.438181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:44.453432 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:44.453384    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m" podStartSLOduration=2.453369626 podStartE2EDuration="2.453369626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 15:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 15:55:44.45205633 +0000 UTC m=+2606.345619969" watchObservedRunningTime="2026-09-19 15:55:44.453369626 +0000 UTC m=+2606.346933266"
Sep 19 15:55:44.781287 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:44.781171    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z2p7w_bb2dc993-51e2-4234-a738-9dd1d61891df/dns/0.log"
Sep 19 15:55:44.802645 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:44.802617    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z2p7w_bb2dc993-51e2-4234-a738-9dd1d61891df/kube-rbac-proxy/0.log"
Sep 19 15:55:44.844298 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:44.844277    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jz2tb_b9ecfb66-d607-4989-8f1e-3651e26cc2a3/dns-node-resolver/0.log"
Sep 19 15:55:45.382693 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:45.382669    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64d48d969c-vl59s_e3436924-04a9-449d-8276-4d71763093dc/registry/0.log"
Sep 19 15:55:45.400914 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:45.400896    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d28d4_500406fa-6c94-45a7-a391-5c758bdd727b/node-ca/0.log"
Sep 19 15:55:47.050549 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:47.050522    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc4565f8d-dnjjq_8393d6f3-9666-4eae-9964-af24387a39fb/kube-auth-proxy/0.log"
Sep 19 15:55:48.101994 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:48.101969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rgzxt_7fcfea47-b51c-41cb-90b0-e656bd0640d5/serve-healthcheck-canary/0.log"
Sep 19 15:55:48.727322 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:48.727296    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ssz8p_b7b96ff1-119d-4684-8be3-c5e2794dc6c1/kube-rbac-proxy/0.log"
Sep 19 15:55:48.745299 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:48.745273    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ssz8p_b7b96ff1-119d-4684-8be3-c5e2794dc6c1/exporter/0.log"
Sep 19 15:55:48.763992 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:48.763974    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ssz8p_b7b96ff1-119d-4684-8be3-c5e2794dc6c1/extractor/0.log"
Sep 19 15:55:50.451776 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:50.451749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-79whr/perf-node-gather-daemonset-ldc2m"
Sep 19 15:55:51.578851 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:51.578824    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d8c7f66c-hq2hz_2eb599ba-05b4-47de-a16a-9fcb1a3a2b37/manager/0.log"
Sep 19 15:55:51.595896 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:51.595876    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d8c7f66c-hq2hz_2eb599ba-05b4-47de-a16a-9fcb1a3a2b37/copy-manifests/0.log"
Sep 19 15:55:53.154013 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:53.153980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7b79856b6-l7kfn_1d1484f1-0345-4655-a9e9-3117ce5f5e6c/manager/0.log"
Sep 19 15:55:53.209113 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:53.209087    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5wc8t_125d5de7-7a89-414e-af80-65298e46035d/openshift-lws-operator/0.log"
Sep 19 15:55:53.639891 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:53.639860    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc4565f8d-dnjjq_8393d6f3-9666-4eae-9964-af24387a39fb/kube-auth-proxy/0.log"
Sep 19 15:55:55.903705 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:55.903672    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-dc4565f8d-dnjjq_8393d6f3-9666-4eae-9964-af24387a39fb/kube-auth-proxy/0.log"
Sep 19 15:55:59.624485 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:59.624457    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2m7_5ab9acdc-6a7c-4407-bfdf-85b2f096455f/kube-multus-additional-cni-plugins/0.log"
Sep 19 15:55:59.642992 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:59.642971    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2m7_5ab9acdc-6a7c-4407-bfdf-85b2f096455f/egress-router-binary-copy/0.log"
Sep 19 15:55:59.662927 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:59.662906    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2m7_5ab9acdc-6a7c-4407-bfdf-85b2f096455f/cni-plugins/0.log"
Sep 19 15:55:59.681934 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:59.681914    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2m7_5ab9acdc-6a7c-4407-bfdf-85b2f096455f/bond-cni-plugin/0.log"
Sep 19 15:55:59.702082 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:59.702063    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2m7_5ab9acdc-6a7c-4407-bfdf-85b2f096455f/routeoverride-cni/0.log"
Sep 19 15:55:59.729391 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:59.729373    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2m7_5ab9acdc-6a7c-4407-bfdf-85b2f096455f/whereabouts-cni-bincopy/0.log"
Sep 19 15:55:59.747648 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:55:59.747624    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2m7_5ab9acdc-6a7c-4407-bfdf-85b2f096455f/whereabouts-cni/0.log"
Sep 19 15:56:00.076989 ip-10-0-141-0 kubenswrapper[2579]: I0919 15:56:00.076958    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dssqn_658ee5aa-e3bf-4397-b6ab-86bdae6793a5/kube-multus/0.log"