Sep 16 13:40:57.004423 ip-10-0-129-226 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 13:40:57.004437 ip-10-0-129-226 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 13:40:57.004447 ip-10-0-129-226 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 13:40:57.004760 ip-10-0-129-226 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 13:41:06.568772 ip-10-0-129-226 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 13:41:06.568784 ip-10-0-129-226 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3971c354b43a423c96b03f566ff60f51 --
Sep 16 13:42:55.858867 ip-10-0-129-226 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 13:42:56.252360 ip-10-0-129-226 kubenswrapper[2548]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:42:56.252360 ip-10-0-129-226 kubenswrapper[2548]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 13:42:56.252360 ip-10-0-129-226 kubenswrapper[2548]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:42:56.252360 ip-10-0-129-226 kubenswrapper[2548]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 13:42:56.252984 ip-10-0-129-226 kubenswrapper[2548]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:42:56.252984 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.252455    2548 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 13:42:56.257242 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257227    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:42:56.257242 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257242    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257246    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257250    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257253    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257256    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257259    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257262    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257265    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257267    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257270    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257272    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257275    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257277    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257280    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257282    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257285    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257287    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257290    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257292    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257300    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:42:56.257308 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257303    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257306    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257310    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257315    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257318    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257322    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257327    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257330    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257333    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257336    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257338    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257341    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257343    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257346    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257349    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257351    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257353    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257356    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257359    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:42:56.257810 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257362    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257364    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257367    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257372    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257375    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257378    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257380    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257383    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257386    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257389    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257391    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257394    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257397    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257399    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257402    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257405    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257408    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257411    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257413    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:42:56.258274 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257416    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257418    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257421    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257424    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257426    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257429    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257431    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257434    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257436    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257439    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257441    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257444    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257446    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257450    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257452    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257455    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257458    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257461    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257463    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257466    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:42:56.258745 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257468    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257471    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257473    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257476    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257478    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257481    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257483    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257486    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257490    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257493    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257496    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257498    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257501    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257503    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257506    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257508    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257510    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257513    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257515    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.257518    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:42:56.259229 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258199    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258205    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258208    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258211    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258213    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258216    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258219    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258222    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258225    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258228    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258231    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258234    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258236    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258239    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258242    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258244    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258247    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258249    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258253    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:42:56.259752 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258258    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258262    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258265    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258267    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258271    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258273    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258276    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258279    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258282    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258285    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258287    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258291    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258293    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258296    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258299    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258302    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258304    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258307    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258310    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258312    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:42:56.260231 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258316    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258318    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258321    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258323    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258326    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258328    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258331    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258333    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258337    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258339    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258342    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258344    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258347    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258350    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258353    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258355    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258358    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258361    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258363    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258366    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:42:56.260816 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258368    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258371    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258373    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258376    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258378    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258381    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258383    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258386    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258388    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258391    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258394    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258396    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258399    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258401    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258404    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258407    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258409    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258412    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258414    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258417    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:42:56.261297 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258420    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258422    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258425    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258428    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258430    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258433    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258436    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258439    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258442    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258444    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258447    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258449    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258453    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258457    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258460    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258463    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258467    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258470    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258474    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:42:56.261802 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258476    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258847    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258851    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258856    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258860    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258863    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258866    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258869    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258872    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258874    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258877    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258879    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258882    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258884    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258887    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258890    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258892    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258895    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258900    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:42:56.262276 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258903    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258905    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258908    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258911    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258913    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258916    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258919    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258921    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258924    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258926    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258929    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258931    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258934    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258936    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258939    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258941    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258944    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258947    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258949    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258952    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:42:56.262744 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258954    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258957    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258960    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258962    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258965    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258967    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258970    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258973    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258975    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258978    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258980    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258987    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258990    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258994    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258996    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.258999    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259001    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259004    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259007    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259009    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:42:56.263235 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259012    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259014    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259017    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259019    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259022    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259024    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259027    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259029    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259032    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259034    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259037    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259040    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259043    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259046    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259048    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259050    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259053    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259056    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259058    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259061    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:42:56.263761 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259063    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259065    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259068    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259074    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259076    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259079    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259081    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259083    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259086    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259089    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259091    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259093    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259096    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259098    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259101    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259104    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259106    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259110    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259113    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259116    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:42:56.264234 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259119    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259183    2548 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259216    2548 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259222    2548 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259226    2548 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259230    2548 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259235    2548 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259239    2548 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259243    2548 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259247    2548 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259251    2548 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259254    2548 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259257    2548 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259260    2548 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259263    2548 flags.go:64] FLAG: --cgroup-root=""
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259268    2548 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259271    2548 flags.go:64] FLAG: --client-ca-file=""
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259273    2548 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259276    2548 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259281    2548 flags.go:64] FLAG: --cluster-domain=""
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259284    2548 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259287    2548 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259290    2548 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259293    2548 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 13:42:56.264762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259297    2548 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259300    2548 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259303    2548 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259306    2548 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259309    2548 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259312    2548 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259315    2548 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259318    2548 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259321    2548 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259324    2548 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259327    2548 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259330    2548 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259333    2548 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259336    2548 flags.go:64] FLAG: --enable-server="true"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259339    2548 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259343    2548 flags.go:64] FLAG: --event-burst="100"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259346    2548 flags.go:64] FLAG: --event-qps="50"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259349    2548 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259352    2548 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259355    2548 flags.go:64] FLAG: --eviction-hard=""
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259358    2548 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259361    2548 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259364    2548 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259368    2548 flags.go:64] FLAG: --eviction-soft=""
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259371    2548 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 13:42:56.265344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259374    2548 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259376    2548 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259384    2548 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259387    2548 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259390    2548 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259393    2548 flags.go:64] FLAG: --feature-gates=""
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259396    2548 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259400    2548 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259403    2548 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259406    2548 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259409    2548 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259412    2548 flags.go:64] FLAG: --help="false"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259415    2548 flags.go:64] FLAG: --hostname-override="ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259418    2548 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259421    2548 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259423    2548 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259427    2548 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259430    2548 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259433    2548 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259435    2548 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259438    2548 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259441    2548 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259444    2548 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259448    2548 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 13:42:56.266314 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259451    2548 flags.go:64] FLAG: --kube-reserved=""
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259454    2548 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259457    2548 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259460    2548 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259463    2548 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259466    2548 flags.go:64] FLAG: --lock-file=""
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259468    2548 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259471    2548 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259474    2548 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259479    2548 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259482    2548 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259485    2548 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259488    2548 flags.go:64] FLAG: --logging-format="text"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259490    2548 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259494    2548 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259496    2548 flags.go:64] FLAG: --manifest-url=""
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259499    2548 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259503    2548 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259506    2548 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259510    2548 flags.go:64] FLAG: --max-pods="110"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259513    2548 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259516    2548 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259519    2548 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259522    2548 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259525    2548 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 13:42:56.267101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259528    2548 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259531    2548 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259537    2548 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259553    2548 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259556    2548 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259560    2548 flags.go:64] FLAG: --pod-cidr=""
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259562    2548 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259569    2548 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259572    2548 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259575    2548 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259578    2548 flags.go:64] FLAG: --port="10250"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259581    2548 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259584    2548 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-038659b3cfff39aa3"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259587    2548 flags.go:64] FLAG: --qos-reserved=""
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259590    2548 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259593    2548 flags.go:64] FLAG: --register-node="true"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259596    2548 flags.go:64] FLAG: --register-with-taints=""
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259599    2548 flags.go:64] FLAG: --registry-burst="10"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259602    2548 flags.go:64] FLAG: --registry-qps="5"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259604    2548 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259607    2548 flags.go:64] FLAG: --reserved-memory=""
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259611    2548 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259613    2548 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259616    2548 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259619    2548 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 13:42:56.267733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259622    2548 flags.go:64] FLAG: --runonce="false"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259625    2548 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259628    2548 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259630    2548 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259633    2548 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259636    2548 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259639    2548 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259641    2548 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259644    2548 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259647    2548 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259650    2548 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259654    2548 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259657    2548 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259661    2548 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259664    2548 flags.go:64] FLAG: --system-cgroups=""
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259668    2548 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259673    2548 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259676    2548 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259679    2548 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259683    2548 flags.go:64] FLAG: --tls-min-version=""
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259686    2548 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259689    2548 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259692    2548 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259694    2548 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259697    2548 flags.go:64] FLAG: --v="2"
Sep 16 13:42:56.268373 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259701    2548 flags.go:64] FLAG: --version="false"
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259707    2548 flags.go:64] FLAG: --vmodule=""
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259711    2548 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.259714    2548 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259826    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259830    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259833    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259836    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259840    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259844    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259847    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259849    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259852    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259855    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259857    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259860    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259862    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259865    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259869    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259872    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:42:56.269024 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259876    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259879    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259882    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259885    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259887    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259890    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259893    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259895    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259898    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259900    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259903    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259906    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259911    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259917    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259921    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259925    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259928    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259932    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259935    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:42:56.269506 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259937    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259940    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259943    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259945    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259948    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259950    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259953    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259955    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259958    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259960    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259963    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259969    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259971    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259975    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259978    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259981    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259983    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259986    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259989    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:42:56.269978 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259992    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259995    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.259998    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260000    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260003    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260005    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260008    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260011    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260013    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260016    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260018    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260021    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260023    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260026    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260029    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260032    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260035    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260038    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260041    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260043    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:42:56.270439 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260046    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260048    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260051    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260053    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260057    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260059    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260063    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260066    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260068    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260071    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260073    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260076    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260079    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260082    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260085    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260087    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260090    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260093    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260095    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260098    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:42:56.270961 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260101    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260104    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260106    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260109    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.260111    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.260119    2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.268100    2548 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.268115    2548 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.269517    2548 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.269526    2548 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269596    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269601    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269605    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269608    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269611    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:42:56.271444 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269614    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269617    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269620    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269623    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269626    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269628    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269631    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269634    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269638    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269642    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269644    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269647    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269650    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269652    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269655    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269658    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269661    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269663    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269666    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269669    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:42:56.271873 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269671    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269674    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269677    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269680    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269683    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269686    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269689    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269693    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269697    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269700    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269703    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269706    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269708    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269711    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269714    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269716    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269719    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269722    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269725    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:42:56.272353 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269728    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269730    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269733    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269736    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269738    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269741    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269744    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269746    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269749    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269751    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269754    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269756    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269760    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269762    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269765    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269767    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269770    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269773    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269776    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:42:56.272926 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269779    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269781    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269784    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269786    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269789    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269792    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269794    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269797    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269799    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269802    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269804    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269807    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269809    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269812    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269814    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269818    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269822    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269825    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269828    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269831    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:42:56.273401 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269833    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269836    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269838    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269841    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269844    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269847    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269850    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269853    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269855    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269858    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269861    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269863    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269866    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269868    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269871    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:42:56.273909 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269874    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.269879    2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269993    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.269999    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270002    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270005    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270008    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270011    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270013    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270016    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270019    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270021    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270024    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270026    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270030    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270032    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:42:56.274279 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270035    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270038    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270041    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270045    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270049    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270052    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270054    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270057    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270060    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270062    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270065    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270068    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270070    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270074    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270078    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270080    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270084    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270086    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270089    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:42:56.274687 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270092    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270095    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270097    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270100    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270102    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270105    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270107    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270110    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270112    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270115    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270117    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270120    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270123    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270126    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270128    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270137    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270140    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270143    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270146    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270148    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:42:56.275133 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270151    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270154    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270156    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270159    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270162    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270165    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270167    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270170    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270172    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270175    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270177    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270181    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270184    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270187    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270190    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270192    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270195    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270197    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270200    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:42:56.275633 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270203    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270205    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270207    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270210    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270213    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270216    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270219    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270222    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270224    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270227    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270229    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270232    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270235    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270238    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270240    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270243    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270246    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270248    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270251    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:42:56.276099 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270253    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270256    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270259    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270261    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270264    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270266    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270268    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:56.270271    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.270276    2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.270375    2548 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.272238    2548 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.273283    2548 server.go:1019] "Starting client certificate rotation"
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.273383    2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 13:42:56.276686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.274072    2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 13:42:56.277005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.276471    2548 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 13:42:56.277577 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.277559    2548 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 13:42:56.288795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.288776    2548 log.go:25] "Validated CRI v1 runtime API"
Sep 16 13:42:56.292151 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.292137    2548 log.go:25] "Validated CRI v1 image API"
Sep 16 13:42:56.292987 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.292969    2548 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 13:42:56.298425 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.298403    2548 fs.go:135] Filesystem UUIDs: map[14770907-7592-4fde-856b-d778618eb26c:/dev/nvme0n1p3 1fe5c9ea-958b-4f0d-9207-f85134b6cec8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 13:42:56.298516 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.298424    2548 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 13:42:56.301259 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.301231    2548 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 13:42:56.304939 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.304834    2548 manager.go:216] Machine: {Timestamp:2026-09-16 13:42:56.303534043 +0000 UTC m=+0.332718826 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099487 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e8a1e2b9ca58211e290c1b7f3e894 SystemUUID:ec2e8a1e-2b9c-a582-11e2-90c1b7f3e894 BootID:3971c354-b43a-423c-96b0-3f566ff60f51 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:91:29:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:91:29:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:a1:bb:e0:b9:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 13:42:56.304939 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.304929    2548 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 13:42:56.305079 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.305028    2548 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 13:42:56.307898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.307872    2548 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 13:42:56.308021 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.307901    2548 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 13:42:56.308582 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.308572    2548 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 13:42:56.308655 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.308585    2548 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 13:42:56.308655 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.308593    2548 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 13:42:56.309173 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.309164    2548 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 13:42:56.309918 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.309909    2548 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 13:42:56.313577 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.313565    2548 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 13:42:56.313692 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.313683    2548 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 13:42:56.313730 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.313711    2548 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 13:42:56.313730 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.313720    2548 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 13:42:56.313803 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.313734    2548 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 13:42:56.313803 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.313743    2548 kubelet.go:398] "Adding apiserver pod source"
Sep 16 13:42:56.313803 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.313751    2548 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 13:42:56.314158 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.314146    2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 13:42:56.314203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.314169    2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 13:42:56.316481 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.316463    2548 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 13:42:56.318015 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.318001    2548 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 13:42:56.318073 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.318032    2548 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 13:42:56.319392 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319382    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 13:42:56.319436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319397    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 13:42:56.319436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319403    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 13:42:56.319436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319409    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 13:42:56.319436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319418    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 13:42:56.319436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319423    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 13:42:56.319436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319437    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 13:42:56.319615 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319443    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 13:42:56.319615 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319450    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 13:42:56.319615 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319455    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 13:42:56.319615 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319463    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 13:42:56.319615 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.319468    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 13:42:56.320368 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.320357    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 13:42:56.320368 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.320366    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 13:42:56.324426 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.324389    2548 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 13:42:56.324426 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.324397    2548 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 13:42:56.324583 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.324468    2548 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 13:42:56.325089 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.325074    2548 server.go:1268] "Started kubelet"
Sep 16 13:42:56.325169 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.325144    2548 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 13:42:56.325211 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.325142    2548 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 13:42:56.325211 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.325199    2548 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 13:42:56.325795 ip-10-0-129-226 systemd[1]: Started Kubernetes Kubelet.
Sep 16 13:42:56.327485 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.327458    2548 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 13:42:56.328915 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.327888    2548 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 13:42:56.329640 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.329620    2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rnlxk"
Sep 16 13:42:56.329708 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.329661    2548 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 13:42:56.331960 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.331942    2548 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 13:42:56.331960 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.331953    2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 13:42:56.332113 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.331142    2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-226.ec2.internal.18d5d0f4b1e8903f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-226.ec2.internal,UID:ip-10-0-129-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-226.ec2.internal,},FirstTimestamp:2026-09-16 13:42:56.325046335 +0000 UTC m=+0.354231118,LastTimestamp:2026-09-16 13:42:56.325046335 +0000 UTC m=+0.354231118,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-226.ec2.internal,}"
Sep 16 13:42:56.332113 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.332070    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:56.332674 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.332490    2548 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 13:42:56.332674 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.332490    2548 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 13:42:56.332674 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.332516    2548 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 13:42:56.332674 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.332608    2548 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 13:42:56.332674 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.332618    2548 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 13:42:56.333240 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.333224    2548 factory.go:55] Registering systemd factory
Sep 16 13:42:56.333289 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.333245    2548 factory.go:223] Registration of the systemd container factory successfully
Sep 16 13:42:56.333613 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.333593    2548 factory.go:153] Registering CRI-O factory
Sep 16 13:42:56.333698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.333617    2548 factory.go:223] Registration of the crio container factory successfully
Sep 16 13:42:56.333751 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.333703    2548 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 13:42:56.333751 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.333721    2548 factory.go:103] Registering Raw factory
Sep 16 13:42:56.333751 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.333732    2548 manager.go:1196] Started watching for new ooms in manager
Sep 16 13:42:56.334339 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.334324    2548 manager.go:349] Starting recovery of all containers
Sep 16 13:42:56.338935 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.338911    2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rnlxk"
Sep 16 13:42:56.339851 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.339823    2548 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 13:42:56.339932 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.339900    2548 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 13:42:56.344992 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.344975    2548 manager.go:354] Recovery completed
Sep 16 13:42:56.349274 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.349259    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:42:56.350968 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.350953    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:42:56.351039 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.350980    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:42:56.351039 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.350994    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:42:56.351488 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.351473    2548 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 13:42:56.351488 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.351486    2548 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 13:42:56.351612 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.351499    2548 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 13:42:56.351612 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.351534    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:42:56.352278 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.352229    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:42:56.352278 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.352252    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:42:56.352278 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.352261    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:42:56.352760 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.352749    2548 policy_none.go:49] "None policy: Start"
Sep 16 13:42:56.352801 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.352766    2548 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 13:42:56.352801 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.352780    2548 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 13:42:56.352999 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.352935    2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-226.ec2.internal.18d5d0f4b3741251  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-226.ec2.internal,UID:ip-10-0-129-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-226.ec2.internal,},FirstTimestamp:2026-09-16 13:42:56.350966353 +0000 UTC m=+0.380151136,LastTimestamp:2026-09-16 13:42:56.350966353 +0000 UTC m=+0.380151136,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-226.ec2.internal,}"
Sep 16 13:42:56.353854 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.353843    2548 policy_none.go:47] "Start"
Sep 16 13:42:56.382123 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.382108    2548 manager.go:337] "Starting Device Plugin manager"
Sep 16 13:42:56.382214 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.382162    2548 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 13:42:56.382214 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.382174    2548 server.go:87] "Starting device plugin registration server"
Sep 16 13:42:56.382399 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.382386    2548 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 13:42:56.382453 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.382400    2548 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 13:42:56.382508 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.382494    2548 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 13:42:56.382653 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.382591    2548 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 13:42:56.382653 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.382603    2548 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 13:42:56.383137 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.383115    2548 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 13:42:56.383217 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.383162    2548 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:56.404004 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.403985    2548 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2581 into the system container \"/system.slice\": <nil>"
Sep 16 13:42:56.482931 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.482906    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:42:56.483796 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.483779    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:42:56.483884 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.483813    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:42:56.483884 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.483828    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:42:56.483884 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.483856    2548 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.491130 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.491117    2548 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.491173 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.491135    2548 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-226.ec2.internal\": node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:56.512939 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.512920    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:56.560449 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.560423    2548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 13:42:56.561567 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.561536    2548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 13:42:56.561645 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.561591    2548 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 13:42:56.561645 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.561612    2548 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 13:42:56.561750 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.561669    2548 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 13:42:56.566798 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.566781    2548 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:42:56.613650 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.613633    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:56.662026 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.662005    2548 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"]
Sep 16 13:42:56.662083 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.662071    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:42:56.663723 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.663705    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:42:56.663793 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.663736    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:42:56.663793 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.663745    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:42:56.665908 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.665897    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:42:56.666019 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.666007    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.666054 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.666034    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:42:56.666893 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.666878    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:42:56.666979 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.666899    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:42:56.666979 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.666904    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:42:56.666979 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.666918    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:42:56.666979 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.666919    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:42:56.666979 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.666955    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:42:56.668935 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.668921    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.669005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.668945    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:42:56.669655 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.669634    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:42:56.669753 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.669664    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:42:56.669753 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.669678    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:42:56.683063 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.683044    2548 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-226.ec2.internal\" not found" node="ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.687940 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.687924    2548 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-226.ec2.internal\" not found" node="ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.713815 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.713799    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:56.734717 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.734698    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02670f36bbb333bf47a0b66bcef8f305-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"02670f36bbb333bf47a0b66bcef8f305\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.734776 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.734721    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02670f36bbb333bf47a0b66bcef8f305-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"02670f36bbb333bf47a0b66bcef8f305\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.734776 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.734736    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/611a46b14cbb4ffe0f7c676f67e60bdf-config\") pod \"kube-apiserver-proxy-ip-10-0-129-226.ec2.internal\" (UID: \"611a46b14cbb4ffe0f7c676f67e60bdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.813900 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.813843    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:56.835321 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.835300    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02670f36bbb333bf47a0b66bcef8f305-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"02670f36bbb333bf47a0b66bcef8f305\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.835372 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.835262    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02670f36bbb333bf47a0b66bcef8f305-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"02670f36bbb333bf47a0b66bcef8f305\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.835410 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.835371    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02670f36bbb333bf47a0b66bcef8f305-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"02670f36bbb333bf47a0b66bcef8f305\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.835410 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.835385    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02670f36bbb333bf47a0b66bcef8f305-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"02670f36bbb333bf47a0b66bcef8f305\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.835410 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.835400    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/611a46b14cbb4ffe0f7c676f67e60bdf-config\") pod \"kube-apiserver-proxy-ip-10-0-129-226.ec2.internal\" (UID: \"611a46b14cbb4ffe0f7c676f67e60bdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.835496 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.835421    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/611a46b14cbb4ffe0f7c676f67e60bdf-config\") pod \"kube-apiserver-proxy-ip-10-0-129-226.ec2.internal\" (UID: \"611a46b14cbb4ffe0f7c676f67e60bdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.914438 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:56.914405    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:56.984911 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.984885    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:56.991445 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:56.991427    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:57.014784 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:57.014760    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:57.115320 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:57.115228    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:58.617490 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:58.617463    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:58.674913 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:58.674879    2548 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 13:42:58.675363 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:58.675048    2548 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 13:42:58.709925 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:58.709904    2548 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:42:58.718361 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:58.718332    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:58.744866 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:58.744839    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 13:37:57 +0000 UTC" deadline="2028-02-22 01:43:56.005765754 +0000 UTC"
Sep 16 13:42:58.744866 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:58.744862    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12564h0m57.260906474s"
Sep 16 13:42:58.819197 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:58.819133    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:58.919608 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:58.919585    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:59.019864 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.019845    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:59.063867 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.063844    2548 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:42:59.120114 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.120096    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found"
Sep 16 13:42:59.165906 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.165726    2548 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:42:59.234129 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.234099    2548 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:59.246187 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.246162    2548 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 13:42:59.247124 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.247109    2548 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"
Sep 16 13:42:59.257022 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.257007    2548 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 13:42:59.302850 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:59.302824    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02670f36bbb333bf47a0b66bcef8f305.slice/crio-d5cf170a0503894862af086fd48e69b7a132589e442db640634a9d3ba5412213 WatchSource:0}: Error finding container d5cf170a0503894862af086fd48e69b7a132589e442db640634a9d3ba5412213: Status 404 returned error can't find the container with id d5cf170a0503894862af086fd48e69b7a132589e442db640634a9d3ba5412213
Sep 16 13:42:59.315877 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.315859    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:42:59.325616 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:42:59.325599    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611a46b14cbb4ffe0f7c676f67e60bdf.slice/crio-534cdcd53f64809fbf4967e1b500235328ce8337538c349358890556c1d03f95 WatchSource:0}: Error finding container 534cdcd53f64809fbf4967e1b500235328ce8337538c349358890556c1d03f95: Status 404 returned error can't find the container with id 534cdcd53f64809fbf4967e1b500235328ce8337538c349358890556c1d03f95
Sep 16 13:42:59.716680 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.716656    2548 apiserver.go:52] "Watching apiserver"
Sep 16 13:42:59.723975 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.723954    2548 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 13:42:59.725125 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.725103    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9kx98","kube-system/konnectivity-agent-mnvnl","kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zrjb8","openshift-dns/node-resolver-xb4zs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal","openshift-multus/multus-additional-cni-plugins-g775r","openshift-network-operator/iptables-alerter-nv897","openshift-ovn-kubernetes/ovnkube-node-xhf7x","kube-system/global-pull-secret-syncer-wqkrw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5","openshift-image-registry/node-ca-sklmz","openshift-multus/multus-jjwls","openshift-multus/network-metrics-daemon-hdb69"]
Sep 16 13:42:59.727928 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.727909    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.729922 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.729904    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.730513 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.730498    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 13:42:59.730589 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.730559    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 13:42:59.730779 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.730754    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 13:42:59.730779 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.730760    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 13:42:59.730920 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.730800    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 13:42:59.731037 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.731019    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ddzvl\""
Sep 16 13:42:59.732677 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.732178    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.732677 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.732455    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7nvjh\""
Sep 16 13:42:59.732677 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.732614    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 13:42:59.734535 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.734516    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:42:59.734701 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.734677    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 13:42:59.734887 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.734867    2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 13:42:59.735610 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.735583    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 13:42:59.735805 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.735790    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xdhdr\""
Sep 16 13:42:59.736018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.736005    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.738286 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.738239    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.738376 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.738288    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:42:59.738440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.738400    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 13:42:59.738484 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.738464    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2b57\""
Sep 16 13:42:59.740460 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.740427    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mrs6j\""
Sep 16 13:42:59.740566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.740496    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.740648 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.740576    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 13:42:59.740710 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.740690    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 13:42:59.742455 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742437    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.742544 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742481    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.742544 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742507    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.742544 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742533    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlsq\" (UniqueName: \"kubernetes.io/projected/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-kube-api-access-2rlsq\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.742680 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742557    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-system-cni-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.742680 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742582    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-etc-kubernetes\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.742680 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742619    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-tuned\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742680 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742637    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:42:59.742680 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742651    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-cni-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.742680 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742675    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-multus-certs\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742695    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f14866dd-faac-43d0-af32-2552b8554302-host-slash\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742719    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-modprobe-d\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742743    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-sys\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742768    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-netns\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742797    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-conf-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742821    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcpk\" (UniqueName: \"kubernetes.io/projected/f14866dd-faac-43d0-af32-2552b8554302-kube-api-access-glcpk\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742840    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysctl-d\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742864    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysctl-conf\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742883    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-run\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742897    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-lib-modules\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742911    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cnibin\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742937    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysconfig\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742957    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-systemd\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742970    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-tmp\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.742972 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742984    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-system-cni-dir\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.742999    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-cni-bin\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743012    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-kubelet\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743026    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-hostroot\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743049    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-kubernetes\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743084    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kj99r\""
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743090    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-var-lib-kubelet\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743130    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpfg\" (UniqueName: \"kubernetes.io/projected/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-kube-api-access-4xpfg\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743160    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-cnibin\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743182    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-cni-binary-copy\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743206    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-host\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743213    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743239    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-socket-dir-parent\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743287    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-k8s-cni-cncf-io\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743317    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwp5\" (UniqueName: \"kubernetes.io/projected/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-kube-api-access-ndwp5\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743348    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743354    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f14866dd-faac-43d0-af32-2552b8554302-iptables-alerter-script\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743384    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.743597 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743407    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-cni-multus\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.744144 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743430    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-daemon-config\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.744144 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.743663    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 13:42:59.744836 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.744823    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:42:59.744899 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.744883    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:42:59.747034 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.747016    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 13:42:59.747126 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.747090    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.747295 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.747283    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-958k5\""
Sep 16 13:42:59.747430 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.747413    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 13:42:59.749285 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.749268    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvl5l\""
Sep 16 13:42:59.749352 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.749269    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 13:42:59.749593 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.749577    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.749666 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.749647    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:42:59.750866 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.750846    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 13:42:59.750866 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.750859    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 13:42:59.751003 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.750891    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 13:42:59.751003 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.750845    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 13:42:59.751003 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.750862    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 13:42:59.751885 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.751861    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.754288 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.754270    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:42:59.754359 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.754331    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:42:59.756517 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.756499    2548 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 13:42:59.757435 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.757417    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 13:42:59.757435 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.757427    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 13:42:59.757609 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.757421    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 13:42:59.757609 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.757474    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nslqn\""
Sep 16 13:42:59.775380 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.775363    2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-874sm"
Sep 16 13:42:59.783283 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.783269    2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-874sm"
Sep 16 13:42:59.835376 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.835358    2548 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 13:42:59.844509 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844488    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq85w\" (UniqueName: \"kubernetes.io/projected/1d932d2e-0c6f-4068-8358-6d091cc9b123-kube-api-access-vq85w\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.844575 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844516    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5-konnectivity-ca\") pod \"konnectivity-agent-mnvnl\" (UID: \"7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5\") " pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:42:59.844575 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844534    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdae81bc-d373-442e-90b2-a48654b28fb6-dbus\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.844575 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844549    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqp8\" (UniqueName: \"kubernetes.io/projected/a4f50b31-1baf-4e33-8e24-9826ee73ace4-kube-api-access-5fqp8\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:42:59.844670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844585    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-socket-dir-parent\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.844670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844618    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-sys-fs\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.844670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844640    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f14866dd-faac-43d0-af32-2552b8554302-iptables-alerter-script\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.844670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844654    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-cni-multus\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.844824 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844676    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-var-lib-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.844824 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844722    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-socket-dir-parent\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.844824 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844732    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-cni-multus\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.844824 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844751    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-node-log\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.844824 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844769    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.844824 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844786    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.844824 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844800    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-system-cni-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.844824 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844821    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-etc-kubernetes\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844837    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844852    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-ovn\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844886    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-tuned\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844913    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-cni-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844912    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-etc-kubernetes\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844935    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844935    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-multus-certs\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844975    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-multus-certs\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844933    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-system-cni-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.844996    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-socket-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845028    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-modprobe-d\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845054    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-sys\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845055    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-cni-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845082    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-netns\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845109    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-kubelet\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845116    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-netns\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845135    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovnkube-config\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.845203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845111    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-sys\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845163    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845172    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-modprobe-d\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845182    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysctl-conf\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845198    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-lib-modules\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845223    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cnibin\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845201    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f14866dd-faac-43d0-af32-2552b8554302-iptables-alerter-script\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845261    2548 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845263    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkt84\" (UniqueName: \"kubernetes.io/projected/aef4300f-e08e-4b1f-948d-ead0bee9764d-kube-api-access-zkt84\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845295    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cnibin\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845310    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-systemd\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845318    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-lib-modules\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845329    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-system-cni-dir\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845330    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysctl-conf\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845359    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-hostroot\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845369    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-systemd\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845373    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-system-cni-dir\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.845944 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845378    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-slash\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845381    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845403    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-hostroot\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845411    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovn-node-metrics-cert\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845439    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-device-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845470    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d932d2e-0c6f-4068-8358-6d091cc9b123-hosts-file\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845490    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-kubernetes\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845504    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpfg\" (UniqueName: \"kubernetes.io/projected/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-kube-api-access-4xpfg\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845518    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-cnibin\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845627    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-kubernetes\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845665    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-cnibin\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845684    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-cni-binary-copy\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845701    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-etc-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845716    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845746    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d932d2e-0c6f-4068-8358-6d091cc9b123-tmp-dir\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845761    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-host\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845775    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-k8s-cni-cncf-io\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.846726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845789    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwp5\" (UniqueName: \"kubernetes.io/projected/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-kube-api-access-ndwp5\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845804    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-systemd\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845826    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5-agent-certs\") pod \"konnectivity-agent-mnvnl\" (UID: \"7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5\") " pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845849    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdae81bc-d373-442e-90b2-a48654b28fb6-kubelet-config\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845849    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-host\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845883    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-run-k8s-cni-cncf-io\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845885    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845943    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-daemon-config\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845969    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-cni-bin\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.845993    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846016    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846038    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846078    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlsq\" (UniqueName: \"kubernetes.io/projected/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-kube-api-access-2rlsq\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846109    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-systemd-units\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846132    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-cni-netd\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846157    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7l7\" (UniqueName: \"kubernetes.io/projected/4ca528a7-467c-4807-8dd2-5019bdc77fff-kube-api-access-np7l7\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846181    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.847326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846191    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-cni-binary-copy\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846206    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-registration-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846230    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvgs\" (UniqueName: \"kubernetes.io/projected/809e016e-5879-42bd-83c4-ea90a7d66cfd-kube-api-access-dhvgs\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846373    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-daemon-config\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846375    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846434    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f14866dd-faac-43d0-af32-2552b8554302-host-slash\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846450    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-conf-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846465    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-run-netns\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846476    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f14866dd-faac-43d0-af32-2552b8554302-host-slash\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846481    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/809e016e-5879-42bd-83c4-ea90a7d66cfd-host\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846507    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcpk\" (UniqueName: \"kubernetes.io/projected/f14866dd-faac-43d0-af32-2552b8554302-kube-api-access-glcpk\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846508    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846523    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysctl-d\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846551    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-run\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846505    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-multus-conf-dir\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846576    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/809e016e-5879-42bd-83c4-ea90a7d66cfd-serviceca\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846586    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysctl-d\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.847773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846600    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysconfig\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846624    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-tmp\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846630    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-run\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846646    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-cni-bin\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846670    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-kubelet\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846683    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-cni-bin\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846685    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-sysconfig\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846696    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-log-socket\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846716    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846721    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-host-var-lib-kubelet\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846751    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846776    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovnkube-script-lib\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846798    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-var-lib-kubelet\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846829    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-env-overrides\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.848261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.846857    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-var-lib-kubelet\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.848652 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.848570    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-tmp\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.848652 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.848593    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-etc-tuned\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.857949 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.857930    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcpk\" (UniqueName: \"kubernetes.io/projected/f14866dd-faac-43d0-af32-2552b8554302-kube-api-access-glcpk\") pod \"iptables-alerter-nv897\" (UID: \"f14866dd-faac-43d0-af32-2552b8554302\") " pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:42:59.857949 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.857942    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlsq\" (UniqueName: \"kubernetes.io/projected/944bfb3d-63e7-449f-9640-ffb1b5ae1a6a-kube-api-access-2rlsq\") pod \"multus-additional-cni-plugins-g775r\" (UID: \"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a\") " pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:42:59.858226 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.858206    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpfg\" (UniqueName: \"kubernetes.io/projected/9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f-kube-api-access-4xpfg\") pod \"tuned-zrjb8\" (UID: \"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f\") " pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:42:59.858313 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.858298    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwp5\" (UniqueName: \"kubernetes.io/projected/571dbb05-2f42-4f69-9fd9-be4c4f7707aa-kube-api-access-ndwp5\") pod \"multus-jjwls\" (UID: \"571dbb05-2f42-4f69-9fd9-be4c4f7707aa\") " pod="openshift-multus/multus-jjwls"
Sep 16 13:42:59.947228 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947169    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-kubelet\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947228 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947199    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovnkube-config\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947228 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947215    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947279    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-kubelet\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947289    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkt84\" (UniqueName: \"kubernetes.io/projected/aef4300f-e08e-4b1f-948d-ead0bee9764d-kube-api-access-zkt84\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947337    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-slash\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947347    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-etc-selinux\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947363    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovn-node-metrics-cert\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947389    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-device-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947391    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-slash\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947412    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d932d2e-0c6f-4068-8358-6d091cc9b123-hosts-file\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.947440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947443    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-etc-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947500    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-etc-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947504    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d932d2e-0c6f-4068-8358-6d091cc9b123-hosts-file\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947498    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947553    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947555    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d932d2e-0c6f-4068-8358-6d091cc9b123-tmp-dir\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947502    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-device-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947610    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-systemd\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947640    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5-agent-certs\") pod \"konnectivity-agent-mnvnl\" (UID: \"7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5\") " pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947666    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdae81bc-d373-442e-90b2-a48654b28fb6-kubelet-config\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947667    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-systemd\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.947795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.947771    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-cni-bin\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.948535 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948514    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-cni-bin\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.948625 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948575    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.948625 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948605    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948638    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-systemd-units\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948669    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-cni-netd\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948698    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7l7\" (UniqueName: \"kubernetes.io/projected/4ca528a7-467c-4807-8dd2-5019bdc77fff-kube-api-access-np7l7\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948726    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948742    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d932d2e-0c6f-4068-8358-6d091cc9b123-tmp-dir\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948750    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-registration-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948783    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvgs\" (UniqueName: \"kubernetes.io/projected/809e016e-5879-42bd-83c4-ea90a7d66cfd-kube-api-access-dhvgs\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948367    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovnkube-config\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948816    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-run-netns\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948610    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdae81bc-d373-442e-90b2-a48654b28fb6-kubelet-config\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.948937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948866    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/809e016e-5879-42bd-83c4-ea90a7d66cfd-host\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948990    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-registration-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949003    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/809e016e-5879-42bd-83c4-ea90a7d66cfd-serviceca\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949045    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-log-socket\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949051    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948889    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-systemd-units\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949079    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948937    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-cni-netd\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949117    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949143    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/809e016e-5879-42bd-83c4-ea90a7d66cfd-host\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.948877    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-run-netns\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949154    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovnkube-script-lib\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949188    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-env-overrides\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949220    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq85w\" (UniqueName: \"kubernetes.io/projected/1d932d2e-0c6f-4068-8358-6d091cc9b123-kube-api-access-vq85w\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949269    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5-konnectivity-ca\") pod \"konnectivity-agent-mnvnl\" (UID: \"7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5\") " pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949303    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdae81bc-d373-442e-90b2-a48654b28fb6-dbus\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949335    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqp8\" (UniqueName: \"kubernetes.io/projected/a4f50b31-1baf-4e33-8e24-9826ee73ace4-kube-api-access-5fqp8\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:42:59.949898 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949363    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-sys-fs\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949498    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949532    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-sys-fs\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949554    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-log-socket\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.948805    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.949638    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs podName:a4f50b31-1baf-4e33-8e24-9826ee73ace4 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:00.449616517 +0000 UTC m=+3.077097883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs") pod "network-metrics-daemon-hdb69" (UID: "a4f50b31-1baf-4e33-8e24-9826ee73ace4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949645    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/809e016e-5879-42bd-83c4-ea90a7d66cfd-serviceca\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949806    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdae81bc-d373-442e-90b2-a48654b28fb6-dbus\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949932    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovnkube-script-lib\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.949985    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-var-lib-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950017    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-node-log\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950052    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950075    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-ovn\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950106    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-socket-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.950126    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.950185    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret podName:cdae81bc-d373-442e-90b2-a48654b28fb6 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:00.450167764 +0000 UTC m=+3.077649127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret") pod "global-pull-secret-syncer-wqkrw" (UID: "cdae81bc-d373-442e-90b2-a48654b28fb6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950227    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aef4300f-e08e-4b1f-948d-ead0bee9764d-socket-dir\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.950698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950272    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-node-log\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.951437 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950326    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-var-lib-openvswitch\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.951437 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950383    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-run-ovn\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.951437 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950446    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ca528a7-467c-4807-8dd2-5019bdc77fff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.951437 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.950720    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5-konnectivity-ca\") pod \"konnectivity-agent-mnvnl\" (UID: \"7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5\") " pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:42:59.951437 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.951315    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5-agent-certs\") pod \"konnectivity-agent-mnvnl\" (UID: \"7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5\") " pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:42:59.951655 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.951625    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ca528a7-467c-4807-8dd2-5019bdc77fff-env-overrides\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.952522 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.952497    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca528a7-467c-4807-8dd2-5019bdc77fff-ovn-node-metrics-cert\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.960896 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.960874    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:42:59.960896 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.960898    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:42:59.961100 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.960912    2548 projected.go:196] Error preparing data for projected volume kube-api-access-lbvfp for pod openshift-network-diagnostics/network-check-target-9kx98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:42:59.961100 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:42:59.960961    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp podName:8e73d474-7109-4801-8634-a735f2dc3ad8 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:00.460946407 +0000 UTC m=+3.088427761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lbvfp" (UniqueName: "kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp") pod "network-check-target-9kx98" (UID: "8e73d474-7109-4801-8634-a735f2dc3ad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:42:59.963197 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.963179    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvgs\" (UniqueName: \"kubernetes.io/projected/809e016e-5879-42bd-83c4-ea90a7d66cfd-kube-api-access-dhvgs\") pod \"node-ca-sklmz\" (UID: \"809e016e-5879-42bd-83c4-ea90a7d66cfd\") " pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:42:59.963346 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.963325    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqp8\" (UniqueName: \"kubernetes.io/projected/a4f50b31-1baf-4e33-8e24-9826ee73ace4-kube-api-access-5fqp8\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:42:59.963422 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.963327    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkt84\" (UniqueName: \"kubernetes.io/projected/aef4300f-e08e-4b1f-948d-ead0bee9764d-kube-api-access-zkt84\") pod \"aws-ebs-csi-driver-node-tvvl5\" (UID: \"aef4300f-e08e-4b1f-948d-ead0bee9764d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:42:59.963622 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.963598    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq85w\" (UniqueName: \"kubernetes.io/projected/1d932d2e-0c6f-4068-8358-6d091cc9b123-kube-api-access-vq85w\") pod \"node-resolver-xb4zs\" (UID: \"1d932d2e-0c6f-4068-8358-6d091cc9b123\") " pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:42:59.963724 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.963689    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7l7\" (UniqueName: \"kubernetes.io/projected/4ca528a7-467c-4807-8dd2-5019bdc77fff-kube-api-access-np7l7\") pod \"ovnkube-node-xhf7x\" (UID: \"4ca528a7-467c-4807-8dd2-5019bdc77fff\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:42:59.967713 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.967650    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" event={"ID":"611a46b14cbb4ffe0f7c676f67e60bdf","Type":"ContainerStarted","Data":"534cdcd53f64809fbf4967e1b500235328ce8337538c349358890556c1d03f95"}
Sep 16 13:42:59.968856 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:42:59.968833    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" event={"ID":"02670f36bbb333bf47a0b66bcef8f305","Type":"ContainerStarted","Data":"d5cf170a0503894862af086fd48e69b7a132589e442db640634a9d3ba5412213"}
Sep 16 13:43:00.040300 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.040274    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g775r"
Sep 16 13:43:00.045216 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.045185    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjwls"
Sep 16 13:43:00.048483 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.048352    2548 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:43:00.050028 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.050003    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944bfb3d_63e7_449f_9640_ffb1b5ae1a6a.slice/crio-f9dbc0ff6c620fdbe0c145cf0e09e097374208a9035805c2ce85ab491bf3aff7 WatchSource:0}: Error finding container f9dbc0ff6c620fdbe0c145cf0e09e097374208a9035805c2ce85ab491bf3aff7: Status 404 returned error can't find the container with id f9dbc0ff6c620fdbe0c145cf0e09e097374208a9035805c2ce85ab491bf3aff7
Sep 16 13:43:00.051950 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.051923    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nv897"
Sep 16 13:43:00.055171 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.055107    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571dbb05_2f42_4f69_9fd9_be4c4f7707aa.slice/crio-c26c0cad62558a8a4e25b4f04f7c7ade8901101dbb0f93acce68f1edb95f070f WatchSource:0}: Error finding container c26c0cad62558a8a4e25b4f04f7c7ade8901101dbb0f93acce68f1edb95f070f: Status 404 returned error can't find the container with id c26c0cad62558a8a4e25b4f04f7c7ade8901101dbb0f93acce68f1edb95f070f
Sep 16 13:43:00.056707 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.056691    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zrjb8"
Sep 16 13:43:00.061536 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.061512    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14866dd_faac_43d0_af32_2552b8554302.slice/crio-b4435c0094705078ea48fbe92ad7e47842eb2b6241935e7f4d0fef16febc5960 WatchSource:0}: Error finding container b4435c0094705078ea48fbe92ad7e47842eb2b6241935e7f4d0fef16febc5960: Status 404 returned error can't find the container with id b4435c0094705078ea48fbe92ad7e47842eb2b6241935e7f4d0fef16febc5960
Sep 16 13:43:00.062110 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.062095    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xb4zs"
Sep 16 13:43:00.066878 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.066855    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1e0a90_2ac8_46f6_a1ff_6f9968400d0f.slice/crio-6075fa318c76c078768f094944fd7cb77ca451a76df3e919d4bf203376e204d9 WatchSource:0}: Error finding container 6075fa318c76c078768f094944fd7cb77ca451a76df3e919d4bf203376e204d9: Status 404 returned error can't find the container with id 6075fa318c76c078768f094944fd7cb77ca451a76df3e919d4bf203376e204d9
Sep 16 13:43:00.067885 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.067860    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sklmz"
Sep 16 13:43:00.072043 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.072013    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d932d2e_0c6f_4068_8358_6d091cc9b123.slice/crio-98f6a32fff157a47d9b750c1a850ed47f775065b674f7c8961300d9b55fd6e62 WatchSource:0}: Error finding container 98f6a32fff157a47d9b750c1a850ed47f775065b674f7c8961300d9b55fd6e62: Status 404 returned error can't find the container with id 98f6a32fff157a47d9b750c1a850ed47f775065b674f7c8961300d9b55fd6e62
Sep 16 13:43:00.075087 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.075064    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:43:00.078989 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.078951    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809e016e_5879_42bd_83c4_ea90a7d66cfd.slice/crio-cd8b1efda3c4494412e7e98b867e0f00726dae4c162648ef162e3b2d18ea54e9 WatchSource:0}: Error finding container cd8b1efda3c4494412e7e98b867e0f00726dae4c162648ef162e3b2d18ea54e9: Status 404 returned error can't find the container with id cd8b1efda3c4494412e7e98b867e0f00726dae4c162648ef162e3b2d18ea54e9
Sep 16 13:43:00.080199 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.080179    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:43:00.084982 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.084966    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5"
Sep 16 13:43:00.088622 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.088599    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb0ef61_c61a_4d07_a6e2_9b4dc5852ef5.slice/crio-2fccf35505f77452d911370d434d0ee24dbedbb19d4f0a828663522050270675 WatchSource:0}: Error finding container 2fccf35505f77452d911370d434d0ee24dbedbb19d4f0a828663522050270675: Status 404 returned error can't find the container with id 2fccf35505f77452d911370d434d0ee24dbedbb19d4f0a828663522050270675
Sep 16 13:43:00.097360 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.097336    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef4300f_e08e_4b1f_948d_ead0bee9764d.slice/crio-a18843908a8531bbd97a2204190d0b2ad07b7dbb0877a875e41c8e227f8143b0 WatchSource:0}: Error finding container a18843908a8531bbd97a2204190d0b2ad07b7dbb0877a875e41c8e227f8143b0: Status 404 returned error can't find the container with id a18843908a8531bbd97a2204190d0b2ad07b7dbb0877a875e41c8e227f8143b0
Sep 16 13:43:00.100462 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:00.100440    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca528a7_467c_4807_8dd2_5019bdc77fff.slice/crio-4ca56a5bbcd2a8560dfd5cd3ed785550494d5bd4f93c1607b7e31c5d30dcb69b WatchSource:0}: Error finding container 4ca56a5bbcd2a8560dfd5cd3ed785550494d5bd4f93c1607b7e31c5d30dcb69b: Status 404 returned error can't find the container with id 4ca56a5bbcd2a8560dfd5cd3ed785550494d5bd4f93c1607b7e31c5d30dcb69b
Sep 16 13:43:00.453907 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.453876    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:00.453907 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.453911    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:00.454149 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.454035    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:00.454149 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.454093    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs podName:a4f50b31-1baf-4e33-8e24-9826ee73ace4 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:01.454076494 +0000 UTC m=+4.081557849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs") pod "network-metrics-daemon-hdb69" (UID: "a4f50b31-1baf-4e33-8e24-9826ee73ace4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:00.454149 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.454038    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:00.454330 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.454173    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret podName:cdae81bc-d373-442e-90b2-a48654b28fb6 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:01.454156449 +0000 UTC m=+4.081637810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret") pod "global-pull-secret-syncer-wqkrw" (UID: "cdae81bc-d373-442e-90b2-a48654b28fb6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:00.554549 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.554517    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:00.554738 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.554720    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:43:00.554804 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.554750    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:43:00.554804 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.554764    2548 projected.go:196] Error preparing data for projected volume kube-api-access-lbvfp for pod openshift-network-diagnostics/network-check-target-9kx98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:00.554884 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.554827    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp podName:8e73d474-7109-4801-8634-a735f2dc3ad8 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:01.554807811 +0000 UTC m=+4.182289167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbvfp" (UniqueName: "kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp") pod "network-check-target-9kx98" (UID: "8e73d474-7109-4801-8634-a735f2dc3ad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:00.786696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.786607    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:37:59 +0000 UTC" deadline="2028-05-25 06:31:45.56751042 +0000 UTC"
Sep 16 13:43:00.786696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.786654    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14800h48m44.780860501s"
Sep 16 13:43:00.963697 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.963662    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:00.963697 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.963689    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:00.963924 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.963798    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:00.963978 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:00.963957    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:00.972071 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.972046    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mnvnl" event={"ID":"7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5","Type":"ContainerStarted","Data":"2fccf35505f77452d911370d434d0ee24dbedbb19d4f0a828663522050270675"}
Sep 16 13:43:00.974941 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.974912    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sklmz" event={"ID":"809e016e-5879-42bd-83c4-ea90a7d66cfd","Type":"ContainerStarted","Data":"cd8b1efda3c4494412e7e98b867e0f00726dae4c162648ef162e3b2d18ea54e9"}
Sep 16 13:43:00.976686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.976657    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xb4zs" event={"ID":"1d932d2e-0c6f-4068-8358-6d091cc9b123","Type":"ContainerStarted","Data":"98f6a32fff157a47d9b750c1a850ed47f775065b674f7c8961300d9b55fd6e62"}
Sep 16 13:43:00.977914 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.977891    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zrjb8" event={"ID":"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f","Type":"ContainerStarted","Data":"6075fa318c76c078768f094944fd7cb77ca451a76df3e919d4bf203376e204d9"}
Sep 16 13:43:00.979085 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.979063    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nv897" event={"ID":"f14866dd-faac-43d0-af32-2552b8554302","Type":"ContainerStarted","Data":"b4435c0094705078ea48fbe92ad7e47842eb2b6241935e7f4d0fef16febc5960"}
Sep 16 13:43:00.980219 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.980200    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g775r" event={"ID":"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a","Type":"ContainerStarted","Data":"f9dbc0ff6c620fdbe0c145cf0e09e097374208a9035805c2ce85ab491bf3aff7"}
Sep 16 13:43:00.981446 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.981428    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5" event={"ID":"aef4300f-e08e-4b1f-948d-ead0bee9764d","Type":"ContainerStarted","Data":"a18843908a8531bbd97a2204190d0b2ad07b7dbb0877a875e41c8e227f8143b0"}
Sep 16 13:43:00.982651 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.982628    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjwls" event={"ID":"571dbb05-2f42-4f69-9fd9-be4c4f7707aa","Type":"ContainerStarted","Data":"c26c0cad62558a8a4e25b4f04f7c7ade8901101dbb0f93acce68f1edb95f070f"}
Sep 16 13:43:00.983863 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:00.983845    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"4ca56a5bbcd2a8560dfd5cd3ed785550494d5bd4f93c1607b7e31c5d30dcb69b"}
Sep 16 13:43:01.460643 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:01.460609    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:01.460821 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:01.460656    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:01.460884 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.460817    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:01.460884 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.460879    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret podName:cdae81bc-d373-442e-90b2-a48654b28fb6 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:03.460861018 +0000 UTC m=+6.088342388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret") pod "global-pull-secret-syncer-wqkrw" (UID: "cdae81bc-d373-442e-90b2-a48654b28fb6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:01.461000 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.460816    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:01.461000 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.460935    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs podName:a4f50b31-1baf-4e33-8e24-9826ee73ace4 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:03.460924554 +0000 UTC m=+6.088405920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs") pod "network-metrics-daemon-hdb69" (UID: "a4f50b31-1baf-4e33-8e24-9826ee73ace4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:01.561384 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:01.561343    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:01.561544 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.561502    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:43:01.561544 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.561525    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:43:01.561544 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.561537    2548 projected.go:196] Error preparing data for projected volume kube-api-access-lbvfp for pod openshift-network-diagnostics/network-check-target-9kx98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:01.561710 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.561594    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp podName:8e73d474-7109-4801-8634-a735f2dc3ad8 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:03.561577497 +0000 UTC m=+6.189058851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbvfp" (UniqueName: "kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp") pod "network-check-target-9kx98" (UID: "8e73d474-7109-4801-8634-a735f2dc3ad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:01.787545 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:01.787464    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:37:59 +0000 UTC" deadline="2028-06-06 00:56:25.990061658 +0000 UTC"
Sep 16 13:43:01.787545 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:01.787504    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15083h13m24.202562765s"
Sep 16 13:43:01.964173 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:01.963716    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:01.964173 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:01.963853    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:01.998828 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:01.998180    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" event={"ID":"611a46b14cbb4ffe0f7c676f67e60bdf","Type":"ContainerStarted","Data":"639d1c307d112a4b3d9da8e520e9b210054363903e7a5e9968117fca8a8fee21"}
Sep 16 13:43:02.963809 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:02.963777    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:02.964485 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:02.963922    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:02.964485 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:02.964313    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:02.964485 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:02.964431    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:03.036963 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:03.036739    2548 generic.go:356] "Generic (PLEG): container finished" podID="02670f36bbb333bf47a0b66bcef8f305" containerID="85c7bfa2b1f9738c5be5422151863c05dab9d42c106ef7504b56ead5c925b712" exitCode=0
Sep 16 13:43:03.037126 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:03.036876    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" event={"ID":"02670f36bbb333bf47a0b66bcef8f305","Type":"ContainerDied","Data":"85c7bfa2b1f9738c5be5422151863c05dab9d42c106ef7504b56ead5c925b712"}
Sep 16 13:43:03.056798 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:03.056738    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" podStartSLOduration=4.056719842 podStartE2EDuration="4.056719842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:02.029401152 +0000 UTC m=+4.656882526" watchObservedRunningTime="2026-09-16 13:43:03.056719842 +0000 UTC m=+5.684201218"
Sep 16 13:43:03.477881 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:03.477728    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:03.477881 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:03.477777    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:03.478096 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.477903    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:03.478096 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.477963    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs podName:a4f50b31-1baf-4e33-8e24-9826ee73ace4 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:07.477945029 +0000 UTC m=+10.105426384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs") pod "network-metrics-daemon-hdb69" (UID: "a4f50b31-1baf-4e33-8e24-9826ee73ace4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:03.478428 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.478349    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:03.478428 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.478401    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret podName:cdae81bc-d373-442e-90b2-a48654b28fb6 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:07.47838654 +0000 UTC m=+10.105867892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret") pod "global-pull-secret-syncer-wqkrw" (UID: "cdae81bc-d373-442e-90b2-a48654b28fb6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:03.578816 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:03.578771    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:03.578983 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.578969    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:43:03.579076 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.578987    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:43:03.579076 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.579000    2548 projected.go:196] Error preparing data for projected volume kube-api-access-lbvfp for pod openshift-network-diagnostics/network-check-target-9kx98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:03.579076 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.579050    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp podName:8e73d474-7109-4801-8634-a735f2dc3ad8 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:07.57903354 +0000 UTC m=+10.206514896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbvfp" (UniqueName: "kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp") pod "network-check-target-9kx98" (UID: "8e73d474-7109-4801-8634-a735f2dc3ad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:03.964084 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:03.963601    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:03.964084 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:03.963721    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:04.045943 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:04.045053    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" event={"ID":"02670f36bbb333bf47a0b66bcef8f305","Type":"ContainerStarted","Data":"bef6f84349a876b8f8bbd52ddb3650b9e7b71b9c08b6e68642cad9a8d9757fc5"}
Sep 16 13:43:04.964378 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:04.964340    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:04.964833 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:04.964340    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:04.964833 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:04.964496    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:04.964833 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:04.964518    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:05.964517 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:05.964482    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:05.964957 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:05.964630    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:06.963756 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:06.963718    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:06.963934 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:06.963855    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:06.963934 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:06.963909    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:06.964060 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:06.964042    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:07.512740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:07.512698    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:07.513184 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:07.512750    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:07.513184 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.512864    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:07.513184 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.512866    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:07.513184 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.512923    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs podName:a4f50b31-1baf-4e33-8e24-9826ee73ace4 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:15.512904616 +0000 UTC m=+18.140385982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs") pod "network-metrics-daemon-hdb69" (UID: "a4f50b31-1baf-4e33-8e24-9826ee73ace4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:07.513184 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.512951    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret podName:cdae81bc-d373-442e-90b2-a48654b28fb6 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:15.512940791 +0000 UTC m=+18.140422147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret") pod "global-pull-secret-syncer-wqkrw" (UID: "cdae81bc-d373-442e-90b2-a48654b28fb6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:07.613923 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:07.613763    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:07.614110 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.613942    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:43:07.614110 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.613963    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:43:07.614110 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.613976    2548 projected.go:196] Error preparing data for projected volume kube-api-access-lbvfp for pod openshift-network-diagnostics/network-check-target-9kx98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:07.614110 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.614038    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp podName:8e73d474-7109-4801-8634-a735f2dc3ad8 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:15.614020104 +0000 UTC m=+18.241501468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbvfp" (UniqueName: "kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp") pod "network-check-target-9kx98" (UID: "8e73d474-7109-4801-8634-a735f2dc3ad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:07.965756 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:07.964596    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:07.965756 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:07.964706    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:08.964539 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:08.964505    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:08.964979 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:08.964505    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:08.964979 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:08.964625    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:08.964979 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:08.964742    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:09.963735 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:09.963702    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:09.963925 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:09.963833    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:10.964401 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:10.964363    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:10.964847 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:10.964404    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:10.964847 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:10.964497    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:10.964847 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:10.964578    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:11.964194 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:11.964163    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:11.964370 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:11.964301    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:12.964225 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:12.964196    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:12.964678 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:12.964339    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:12.964678 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:12.964386    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:12.964678 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:12.964488    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:13.963938 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:13.963903    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:13.964079 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:13.964033    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:14.963932 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:14.963901    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:14.964399 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:14.963911    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:14.964399 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:14.963996    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:14.964399 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:14.964123    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:15.573989 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:15.573952    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:15.573989 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:15.573994    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:15.574351 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.574107    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:15.574351 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.574112    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:15.574351 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.574165    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs podName:a4f50b31-1baf-4e33-8e24-9826ee73ace4 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:31.574145321 +0000 UTC m=+34.201626675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs") pod "network-metrics-daemon-hdb69" (UID: "a4f50b31-1baf-4e33-8e24-9826ee73ace4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:15.574351 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.574183    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret podName:cdae81bc-d373-442e-90b2-a48654b28fb6 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:31.57417421 +0000 UTC m=+34.201655568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret") pod "global-pull-secret-syncer-wqkrw" (UID: "cdae81bc-d373-442e-90b2-a48654b28fb6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:15.675140 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:15.675103    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:15.675309 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.675232    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:43:15.675309 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.675265    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:43:15.675309 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.675277    2548 projected.go:196] Error preparing data for projected volume kube-api-access-lbvfp for pod openshift-network-diagnostics/network-check-target-9kx98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:15.675438 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.675323    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp podName:8e73d474-7109-4801-8634-a735f2dc3ad8 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:31.675310272 +0000 UTC m=+34.302791627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbvfp" (UniqueName: "kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp") pod "network-check-target-9kx98" (UID: "8e73d474-7109-4801-8634-a735f2dc3ad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:15.964032 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:15.963994    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:15.964498 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:15.964112    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:16.964342 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:16.964305    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:16.964826 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:16.964351    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:16.964826 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:16.964447    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:16.964826 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:16.964541    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:17.964278 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:17.964222    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:17.964430 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:17.964333    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:18.964577 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:18.964411    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:18.965285 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:18.964467    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:18.965285 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:18.964659    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:18.965285 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:18.964766    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:19.077470 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.077388    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sklmz" event={"ID":"809e016e-5879-42bd-83c4-ea90a7d66cfd","Type":"ContainerStarted","Data":"36447617b7be9444900e023eb65927ccfbbb67ccdf044420160795c5f96d136d"}
Sep 16 13:43:19.078745 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.078721    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xb4zs" event={"ID":"1d932d2e-0c6f-4068-8358-6d091cc9b123","Type":"ContainerStarted","Data":"0fc8909989527f4ef68616f5b1e5226139c8bbe2353c98b891993af43f463514"}
Sep 16 13:43:19.080071 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.080048    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zrjb8" event={"ID":"9c1e0a90-2ac8-46f6-a1ff-6f9968400d0f","Type":"ContainerStarted","Data":"ad2c1a897e6c07ca518bab08de42f6e8ed62825c658eaf21dbba541e3b0a44b9"}
Sep 16 13:43:19.081603 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.081580    2548 generic.go:356] "Generic (PLEG): container finished" podID="944bfb3d-63e7-449f-9640-ffb1b5ae1a6a" containerID="67832c9518af3d05ce42a0f4afca2ee932921cf7ef24e16a2530adb3505b855a" exitCode=0
Sep 16 13:43:19.081713 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.081662    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g775r" event={"ID":"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a","Type":"ContainerDied","Data":"67832c9518af3d05ce42a0f4afca2ee932921cf7ef24e16a2530adb3505b855a"}
Sep 16 13:43:19.083221 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.083200    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5" event={"ID":"aef4300f-e08e-4b1f-948d-ead0bee9764d","Type":"ContainerStarted","Data":"318590785c421caacb5ebd73453e1958e7507040f873aafd040f24136e1aeddf"}
Sep 16 13:43:19.084746 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.084726    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjwls" event={"ID":"571dbb05-2f42-4f69-9fd9-be4c4f7707aa","Type":"ContainerStarted","Data":"dd0f38d8c1969ef2051f4c07254c5e94dcba7a3c38c691a8229733a6a97a9f65"}
Sep 16 13:43:19.087329 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.087314    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:43:19.087651 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.087634    2548 generic.go:356] "Generic (PLEG): container finished" podID="4ca528a7-467c-4807-8dd2-5019bdc77fff" containerID="a9b02232fdbf255d53e3dc2a14b65f66001cc74f5e723d31c340f7b6b18a85ad" exitCode=1
Sep 16 13:43:19.087708 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.087688    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"3891ff3b3bc1e667b83a47c9d45ec119e09a4fba2d46208d702d1b2b7b0a5e96"}
Sep 16 13:43:19.087749 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.087738    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"118f3f79432a6d595f839c0c43dd14e74c222d21f4b0a80971df0c6959bd886e"}
Sep 16 13:43:19.087800 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.087785    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"97e2bdab2f64a237395a7892b69ff2bcd9775b470c696904951451aaa2952e67"}
Sep 16 13:43:19.087842 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.087832    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"87ee90c27d4c1002831100e6dcd6b75cab12d963f98943fda6ebdd4b9960712e"}
Sep 16 13:43:19.087880 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.087859    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerDied","Data":"a9b02232fdbf255d53e3dc2a14b65f66001cc74f5e723d31c340f7b6b18a85ad"}
Sep 16 13:43:19.087924 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.087905    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"6d1e1f00f7a7f6dc8cbbc35565fca0edbcc994f8e089ceb27107ef9a36a02863"}
Sep 16 13:43:19.088986 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.088970    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mnvnl" event={"ID":"7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5","Type":"ContainerStarted","Data":"dba6e4f57041582d51721b909e3122a67f2bbd0b53ae60f8ef734cfc23f7e7af"}
Sep 16 13:43:19.096727 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.096688    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" podStartSLOduration=20.096677375 podStartE2EDuration="20.096677375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:04.066086678 +0000 UTC m=+6.693568074" watchObservedRunningTime="2026-09-16 13:43:19.096677375 +0000 UTC m=+21.724158767"
Sep 16 13:43:19.097031 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.097012    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sklmz" podStartSLOduration=4.009899903 podStartE2EDuration="22.097008606s" totalImagesPullingTime="18.087108703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:19.096463125 +0000 UTC m=+21.723944524" watchObservedRunningTime="2026-09-16 13:43:19.097008606 +0000 UTC m=+21.724489979"
Sep 16 13:43:19.136464 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.136432    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jjwls" podStartSLOduration=3.487393159 podStartE2EDuration="22.136419306s" totalImagesPullingTime="18.649026147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:19.136145603 +0000 UTC m=+21.763626977" watchObservedRunningTime="2026-09-16 13:43:19.136419306 +0000 UTC m=+21.763900680"
Sep 16 13:43:19.171731 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.171694    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mnvnl" podStartSLOduration=4.092420636 podStartE2EDuration="22.171678509s" totalImagesPullingTime="18.079257873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:19.171606969 +0000 UTC m=+21.799088342" watchObservedRunningTime="2026-09-16 13:43:19.171678509 +0000 UTC m=+21.799159885"
Sep 16 13:43:19.171882 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.171848    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xb4zs" podStartSLOduration=4.05174015 podStartE2EDuration="22.171843482s" totalImagesPullingTime="18.120103332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:19.154857992 +0000 UTC m=+21.782339368" watchObservedRunningTime="2026-09-16 13:43:19.171843482 +0000 UTC m=+21.799324855"
Sep 16 13:43:19.189598 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.189561    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zrjb8" podStartSLOduration=4.063341619 podStartE2EDuration="22.18954952s" totalImagesPullingTime="18.126207901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:19.189446461 +0000 UTC m=+21.816927836" watchObservedRunningTime="2026-09-16 13:43:19.18954952 +0000 UTC m=+21.817030894"
Sep 16 13:43:19.687957 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.687903    2548 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 13:43:19.797205 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.797084    2548 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T13:43:19.687921148Z","UUID":"abb0c53c-9118-41b9-9d50-153abd459236","Handler":null,"Name":"","Endpoint":""}
Sep 16 13:43:19.798400 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.798381    2548 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 13:43:19.798495 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.798410    2548 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 13:43:19.963969 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:19.963942    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:19.964098 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:19.964039    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:20.092670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:20.092587    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nv897" event={"ID":"f14866dd-faac-43d0-af32-2552b8554302","Type":"ContainerStarted","Data":"f27fbe46887c9e510861c29c08cb6350ec7fc48c75c24eaaf4af5234ab583989"}
Sep 16 13:43:20.094942 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:20.094910    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5" event={"ID":"aef4300f-e08e-4b1f-948d-ead0bee9764d","Type":"ContainerStarted","Data":"d59c794061a886406208ce571f0c79263fefa33b82c82a29cf44befec2745bba"}
Sep 16 13:43:20.110219 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:20.110177    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nv897" podStartSLOduration=5.003566154 podStartE2EDuration="23.110163771s" totalImagesPullingTime="18.106597617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:20.11000372 +0000 UTC m=+22.737485094" watchObservedRunningTime="2026-09-16 13:43:20.110163771 +0000 UTC m=+22.737645157"
Sep 16 13:43:20.196414 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:20.196374    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:43:20.668319 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:20.668286    2548 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mnvnl_7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:20.771907 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:20.771842    2548 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mnvnl_7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:20.963876 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:20.963844    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:20.963989 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:20.963844    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:20.963989 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:20.963962    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:20.964107 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:20.964008    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:21.100120 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:21.100080    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5" event={"ID":"aef4300f-e08e-4b1f-948d-ead0bee9764d","Type":"ContainerStarted","Data":"afae92b5aad2b9620efc759ac65e6ec02d9148f7e1a85eaa350d1306d84625f1"}
Sep 16 13:43:21.103773 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:21.103752    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:43:21.104306 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:21.104279    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"2e28f7f3ecc9dee29aa51ca797e9e47ba46d3674eee5e546a9db729652cc934f"}
Sep 16 13:43:21.121598 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:21.121557    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tvvl5" podStartSLOduration=2.7925491669999998 podStartE2EDuration="23.121542413s" totalImagesPullingTime="20.328993246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:21.120878901 +0000 UTC m=+23.748360277" watchObservedRunningTime="2026-09-16 13:43:21.121542413 +0000 UTC m=+23.749023787"
Sep 16 13:43:21.964162 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:21.964120    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:21.964474 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:21.964239    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:22.931017 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:22.930978    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:43:22.931590 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:22.931572    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:43:22.964061 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:22.964031    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:22.964207 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:22.964150    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:22.964207 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:22.964201    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:22.964364 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:22.964276    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:23.109386 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:23.109188    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mnvnl"
Sep 16 13:43:23.963909 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:23.963869    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:23.964320 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:23.964001    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:24.964234 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:24.964197    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:24.964937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:24.964197    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:24.964937 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:24.964343    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:24.964937 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:24.964448    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:25.115950 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.115926    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:43:25.116327 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.116306    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"0cd6c11297445821358af37b0ccdd395a04ef046247e322a7ba2ee99c68f28d4"}
Sep 16 13:43:25.116567 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.116544    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:43:25.116696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.116617    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:43:25.116755 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.116715    2548 scope.go:117] "RemoveContainer" containerID="a9b02232fdbf255d53e3dc2a14b65f66001cc74f5e723d31c340f7b6b18a85ad"
Sep 16 13:43:25.118440 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.118416    2548 generic.go:356] "Generic (PLEG): container finished" podID="944bfb3d-63e7-449f-9640-ffb1b5ae1a6a" containerID="dae2b55872b05dba95fb133598c4984e13834bb68e7b36f50b61ff562ae80667" exitCode=0
Sep 16 13:43:25.118544 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.118457    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g775r" event={"ID":"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a","Type":"ContainerDied","Data":"dae2b55872b05dba95fb133598c4984e13834bb68e7b36f50b61ff562ae80667"}
Sep 16 13:43:25.132298 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.132282    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:43:25.132810 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.132795    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:43:25.964262 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:25.964105    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:25.964666 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:25.964373    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:26.124626 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.124604    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:43:26.124963 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.124936    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" event={"ID":"4ca528a7-467c-4807-8dd2-5019bdc77fff","Type":"ContainerStarted","Data":"5fcc9c295c336cc9578a3c39c42ab44bedde1b46e880974de57468330ce089d4"}
Sep 16 13:43:26.125059 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.125027    2548 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 13:43:26.160335 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.160199    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x" podStartSLOduration=10.058163116 podStartE2EDuration="28.160184353s" totalImagesPullingTime="18.102021237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:26.158148932 +0000 UTC m=+28.785630307" watchObservedRunningTime="2026-09-16 13:43:26.160184353 +0000 UTC m=+28.787665726"
Sep 16 13:43:26.406628 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.406579    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wqkrw"]
Sep 16 13:43:26.406769 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.406754    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:26.406899 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:26.406843    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:26.408470 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.408451    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hdb69"]
Sep 16 13:43:26.408553 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.408541    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:26.408640 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:26.408625    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:26.419108 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.418700    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9kx98"]
Sep 16 13:43:26.419108 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:26.418797    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:26.419108 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:26.418883    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:27.129212 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:27.129179    2548 generic.go:356] "Generic (PLEG): container finished" podID="944bfb3d-63e7-449f-9640-ffb1b5ae1a6a" containerID="2a875a0bb0f9976f8eeae8366d67db9c6f1b2958b8e7d6c4eb054f30b0c84c25" exitCode=0
Sep 16 13:43:27.129686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:27.129276    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g775r" event={"ID":"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a","Type":"ContainerDied","Data":"2a875a0bb0f9976f8eeae8366d67db9c6f1b2958b8e7d6c4eb054f30b0c84c25"}
Sep 16 13:43:27.129686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:27.129370    2548 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 13:43:27.604480 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:27.604453    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:43:27.965667 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:27.965131    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:27.965667 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:27.965232    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:27.965896 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:27.965762    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:27.965896 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:27.965875    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:27.966165 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:27.966016    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:27.966165 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:27.966131    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:29.136106 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:29.135937    2548 generic.go:356] "Generic (PLEG): container finished" podID="944bfb3d-63e7-449f-9640-ffb1b5ae1a6a" containerID="d5078e4c9512ef7b183646db9926573ffe35d1020a77ec2a7f7c1486a5874988" exitCode=0
Sep 16 13:43:29.136106 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:29.136022    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g775r" event={"ID":"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a","Type":"ContainerDied","Data":"d5078e4c9512ef7b183646db9926573ffe35d1020a77ec2a7f7c1486a5874988"}
Sep 16 13:43:29.964492 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:29.964458    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:29.964649 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:29.964467    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:29.964649 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:29.964562    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqkrw" podUID="cdae81bc-d373-442e-90b2-a48654b28fb6"
Sep 16 13:43:29.964649 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:29.964620    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hdb69" podUID="a4f50b31-1baf-4e33-8e24-9826ee73ace4"
Sep 16 13:43:29.964761 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:29.964654    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:29.964761 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:29.964698    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9kx98" podUID="8e73d474-7109-4801-8634-a735f2dc3ad8"
Sep 16 13:43:30.183448 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.183410    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeReady"
Sep 16 13:43:30.183835 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.183550    2548 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 13:43:30.240262 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.240170    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7446ffdbbc-qwcxh"]
Sep 16 13:43:30.251788 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.251764    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.258762 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.258503    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s4xvt\""
Sep 16 13:43:30.260172 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.259520    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gh2dv"]
Sep 16 13:43:30.261579 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.261469    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 13:43:30.261701 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.261579    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 13:43:30.265142 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.265122    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 13:43:30.265333 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.265318    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 13:43:30.268802 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.268783    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bxzwx"]
Sep 16 13:43:30.268921 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.268905    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gh2dv"
Sep 16 13:43:30.275187 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.275163    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.276901 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.276880    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7446ffdbbc-qwcxh"]
Sep 16 13:43:30.277377 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.277353    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n4dzm\""
Sep 16 13:43:30.277478 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.277365    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 13:43:30.277478 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.277427    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 13:43:30.277581 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.277514    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 13:43:30.278157 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.278126    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 13:43:30.278316 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.278200    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kqx7z\""
Sep 16 13:43:30.278316 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.278212    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 13:43:30.282122 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.282090    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bxzwx"]
Sep 16 13:43:30.282790 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.282772    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gh2dv"]
Sep 16 13:43:30.382938 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.382910    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cca1368-1fe3-483a-a299-13b6f79aa8ac-cert\") pod \"ingress-canary-gh2dv\" (UID: \"5cca1368-1fe3-483a-a299-13b6f79aa8ac\") " pod="openshift-ingress-canary/ingress-canary-gh2dv"
Sep 16 13:43:30.383109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.382972    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f7qk\" (UniqueName: \"kubernetes.io/projected/ef388d95-2cfe-4771-aa31-0128b60b0723-kube-api-access-7f7qk\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.383109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.382996    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-bound-sa-token\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.383109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383035    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef388d95-2cfe-4771-aa31-0128b60b0723-tmp-dir\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.383109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383061    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-registry-certificates\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.383109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383083    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-trusted-ca\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.383361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383136    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef388d95-2cfe-4771-aa31-0128b60b0723-config-volume\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.383361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383163    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.383361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383185    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51726060-7491-449a-9591-9cc5c8bab5b1-ca-trust-extracted\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.383361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383243    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-image-registry-private-configuration\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.383361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383295    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.383361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383312    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gtrb\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-kube-api-access-5gtrb\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.383361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383341    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-installation-pull-secrets\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.383612 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.383405    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdf9h\" (UniqueName: \"kubernetes.io/projected/5cca1368-1fe3-483a-a299-13b6f79aa8ac-kube-api-access-zdf9h\") pod \"ingress-canary-gh2dv\" (UID: \"5cca1368-1fe3-483a-a299-13b6f79aa8ac\") " pod="openshift-ingress-canary/ingress-canary-gh2dv"
Sep 16 13:43:30.484281 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484224    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef388d95-2cfe-4771-aa31-0128b60b0723-tmp-dir\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.484281 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484280    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-registry-certificates\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.484570 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484305    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-trusted-ca\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.484570 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484326    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef388d95-2cfe-4771-aa31-0128b60b0723-config-volume\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.484570 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484352    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.484570 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484377    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51726060-7491-449a-9591-9cc5c8bab5b1-ca-trust-extracted\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.484570 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484480    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-image-registry-private-configuration\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.484570 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.484504    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 13:43:30.484570 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.484522    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7446ffdbbc-qwcxh: secret "image-registry-tls" not found
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.484585    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls podName:51726060-7491-449a-9591-9cc5c8bab5b1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:30.984563652 +0000 UTC m=+33.612045022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls") pod "image-registry-7446ffdbbc-qwcxh" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1") : secret "image-registry-tls" not found
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.484599    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484631    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef388d95-2cfe-4771-aa31-0128b60b0723-tmp-dir\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.484659    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls podName:ef388d95-2cfe-4771-aa31-0128b60b0723 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:30.984643759 +0000 UTC m=+33.612125127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls") pod "dns-default-bxzwx" (UID: "ef388d95-2cfe-4771-aa31-0128b60b0723") : secret "dns-default-metrics-tls" not found
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484756    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51726060-7491-449a-9591-9cc5c8bab5b1-ca-trust-extracted\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484518    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484806    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtrb\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-kube-api-access-5gtrb\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484838    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-installation-pull-secrets\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484877    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdf9h\" (UniqueName: \"kubernetes.io/projected/5cca1368-1fe3-483a-a299-13b6f79aa8ac-kube-api-access-zdf9h\") pod \"ingress-canary-gh2dv\" (UID: \"5cca1368-1fe3-483a-a299-13b6f79aa8ac\") " pod="openshift-ingress-canary/ingress-canary-gh2dv"
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484914    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cca1368-1fe3-483a-a299-13b6f79aa8ac-cert\") pod \"ingress-canary-gh2dv\" (UID: \"5cca1368-1fe3-483a-a299-13b6f79aa8ac\") " pod="openshift-ingress-canary/ingress-canary-gh2dv"
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484959    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f7qk\" (UniqueName: \"kubernetes.io/projected/ef388d95-2cfe-4771-aa31-0128b60b0723-kube-api-access-7f7qk\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.485029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.484986    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-bound-sa-token\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.485757 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.485207    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-registry-certificates\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.485757 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.485283    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef388d95-2cfe-4771-aa31-0128b60b0723-config-volume\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.485757 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.485362    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-trusted-ca\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.489382 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.489363    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-installation-pull-secrets\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.489382 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.489388    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-image-registry-private-configuration\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.489382 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.489399    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cca1368-1fe3-483a-a299-13b6f79aa8ac-cert\") pod \"ingress-canary-gh2dv\" (UID: \"5cca1368-1fe3-483a-a299-13b6f79aa8ac\") " pod="openshift-ingress-canary/ingress-canary-gh2dv"
Sep 16 13:43:30.498478 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.498413    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-bound-sa-token\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.498581 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.498484    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f7qk\" (UniqueName: \"kubernetes.io/projected/ef388d95-2cfe-4771-aa31-0128b60b0723-kube-api-access-7f7qk\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.498639 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.498590    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtrb\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-kube-api-access-5gtrb\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.498696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.498675    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdf9h\" (UniqueName: \"kubernetes.io/projected/5cca1368-1fe3-483a-a299-13b6f79aa8ac-kube-api-access-zdf9h\") pod \"ingress-canary-gh2dv\" (UID: \"5cca1368-1fe3-483a-a299-13b6f79aa8ac\") " pod="openshift-ingress-canary/ingress-canary-gh2dv"
Sep 16 13:43:30.576905 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.576870    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gh2dv"
Sep 16 13:43:30.885008 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.884977    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gh2dv"]
Sep 16 13:43:30.889053 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:30.889024    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cca1368_1fe3_483a_a299_13b6f79aa8ac.slice/crio-a2af81ac96cfbbae76b4d7270c8c621f26c905a1571c463c6305997b7a1a4cec WatchSource:0}: Error finding container a2af81ac96cfbbae76b4d7270c8c621f26c905a1571c463c6305997b7a1a4cec: Status 404 returned error can't find the container with id a2af81ac96cfbbae76b4d7270c8c621f26c905a1571c463c6305997b7a1a4cec
Sep 16 13:43:30.989467 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.989431    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:30.989646 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:30.989498    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:30.989646 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.989614    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 13:43:30.989646 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.989636    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7446ffdbbc-qwcxh: secret "image-registry-tls" not found
Sep 16 13:43:30.989814 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.989656    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:43:30.989814 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.989710    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls podName:51726060-7491-449a-9591-9cc5c8bab5b1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:31.989689359 +0000 UTC m=+34.617170718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls") pod "image-registry-7446ffdbbc-qwcxh" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1") : secret "image-registry-tls" not found
Sep 16 13:43:30.989814 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:30.989729    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls podName:ef388d95-2cfe-4771-aa31-0128b60b0723 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:31.98971904 +0000 UTC m=+34.617200399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls") pod "dns-default-bxzwx" (UID: "ef388d95-2cfe-4771-aa31-0128b60b0723") : secret "dns-default-metrics-tls" not found
Sep 16 13:43:31.144691 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.144603    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gh2dv" event={"ID":"5cca1368-1fe3-483a-a299-13b6f79aa8ac","Type":"ContainerStarted","Data":"a2af81ac96cfbbae76b4d7270c8c621f26c905a1571c463c6305997b7a1a4cec"}
Sep 16 13:43:31.594379 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.594296    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:31.594379 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.594355    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:31.595237 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.594463    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:31.595237 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.594544    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret podName:cdae81bc-d373-442e-90b2-a48654b28fb6 nodeName:}" failed. No retries permitted until 2026-09-16 13:44:03.594522392 +0000 UTC m=+66.222003757 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret") pod "global-pull-secret-syncer-wqkrw" (UID: "cdae81bc-d373-442e-90b2-a48654b28fb6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:43:31.595237 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.594469    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:31.595237 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.594624    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs podName:a4f50b31-1baf-4e33-8e24-9826ee73ace4 nodeName:}" failed. No retries permitted until 2026-09-16 13:44:03.59460661 +0000 UTC m=+66.222087969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs") pod "network-metrics-daemon-hdb69" (UID: "a4f50b31-1baf-4e33-8e24-9826ee73ace4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:43:31.695661 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.695586    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:31.695856 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.695780    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:43:31.695856 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.695805    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:43:31.695856 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.695824    2548 projected.go:196] Error preparing data for projected volume kube-api-access-lbvfp for pod openshift-network-diagnostics/network-check-target-9kx98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:31.696020 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.695893    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp podName:8e73d474-7109-4801-8634-a735f2dc3ad8 nodeName:}" failed. No retries permitted until 2026-09-16 13:44:03.695874714 +0000 UTC m=+66.323356070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbvfp" (UniqueName: "kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp") pod "network-check-target-9kx98" (UID: "8e73d474-7109-4801-8634-a735f2dc3ad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:43:31.839259 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.839212    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-shkw6"]
Sep 16 13:43:31.841558 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.841535    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:31.844522 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.844455    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dks\""
Sep 16 13:43:31.845346 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.845317    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 13:43:31.845617 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.845596    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 13:43:31.846273 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.846090    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 13:43:31.846703 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.846528    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 13:43:31.860507 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.860487    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-shkw6"]
Sep 16 13:43:31.964349 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.964310    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:43:31.964349 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.964334    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:43:31.964568 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.964313    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:43:31.967477 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.967399    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 13:43:31.967477 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.967448    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7xrvw\""
Sep 16 13:43:31.967477 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.967454    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 13:43:31.967477 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.967473    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4knh2\""
Sep 16 13:43:31.967737 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.967487    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 13:43:31.967737 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.967506    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 13:43:31.997527 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.997479    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:31.997644 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.997564    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:31.997644 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.997597    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-crio-socket\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:31.997644 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.997624    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdqtk\" (UniqueName: \"kubernetes.io/projected/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-api-access-gdqtk\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:31.997802 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.997744    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:31.997856 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.997798    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-data-volume\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:31.997856 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.997826    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 13:43:31.997856 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:31.997830    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:31.997856 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.997842    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7446ffdbbc-qwcxh: secret "image-registry-tls" not found
Sep 16 13:43:31.998051 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.997899    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls podName:51726060-7491-449a-9591-9cc5c8bab5b1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:33.997880658 +0000 UTC m=+36.625362026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls") pod "image-registry-7446ffdbbc-qwcxh" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1") : secret "image-registry-tls" not found
Sep 16 13:43:31.998051 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.997935    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:43:31.998051 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:31.997978    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls podName:ef388d95-2cfe-4771-aa31-0128b60b0723 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:33.997963885 +0000 UTC m=+36.625445243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls") pod "dns-default-bxzwx" (UID: "ef388d95-2cfe-4771-aa31-0128b60b0723") : secret "dns-default-metrics-tls" not found
Sep 16 13:43:32.098609 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.098528    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.098761 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.098618    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.098761 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.098647    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-crio-socket\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.098761 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.098665    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdqtk\" (UniqueName: \"kubernetes.io/projected/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-api-access-gdqtk\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.098761 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.098744    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-data-volume\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.099153 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:32.099116    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:32.099214 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:32.099161    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:43:32.099214 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.099168    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-crio-socket\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.099214 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:32.099196    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:32.59917524 +0000 UTC m=+35.226656606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:32.099385 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.099114    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-data-volume\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.099385 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:32.099214    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:32.599203946 +0000 UTC m=+35.226685298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : configmap "kube-rbac-proxy" not found
Sep 16 13:43:32.114916 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.114884    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdqtk\" (UniqueName: \"kubernetes.io/projected/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-api-access-gdqtk\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.536307 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.536233    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz"]
Sep 16 13:43:32.539059 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.539034    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz"
Sep 16 13:43:32.541790 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.541765    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 13:43:32.542753 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.542731    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 13:43:32.542862 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.542734    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pnbnp\""
Sep 16 13:43:32.551953 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.551930    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz"]
Sep 16 13:43:32.604628 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.604596    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.605023 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.604706    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:32.605023 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:32.604715    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:43:32.605023 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:32.604781    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:33.60476373 +0000 UTC m=+36.232245094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : configmap "kube-rbac-proxy" not found
Sep 16 13:43:32.605023 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:32.604836    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:32.605023 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:32.604872    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:33.604860308 +0000 UTC m=+36.232341666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:32.705361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.705322    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgqs\" (UniqueName: \"kubernetes.io/projected/0aa72ef3-33ed-486b-b91f-648f4be57a47-kube-api-access-kmgqs\") pod \"migrator-8b4d697c5-ll8lz\" (UID: \"0aa72ef3-33ed-486b-b91f-648f4be57a47\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz"
Sep 16 13:43:32.806554 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.806472    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgqs\" (UniqueName: \"kubernetes.io/projected/0aa72ef3-33ed-486b-b91f-648f4be57a47-kube-api-access-kmgqs\") pod \"migrator-8b4d697c5-ll8lz\" (UID: \"0aa72ef3-33ed-486b-b91f-648f4be57a47\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz"
Sep 16 13:43:32.817916 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.817890    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgqs\" (UniqueName: \"kubernetes.io/projected/0aa72ef3-33ed-486b-b91f-648f4be57a47-kube-api-access-kmgqs\") pod \"migrator-8b4d697c5-ll8lz\" (UID: \"0aa72ef3-33ed-486b-b91f-648f4be57a47\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz"
Sep 16 13:43:32.850811 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:32.850779    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz"
Sep 16 13:43:33.380598 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:33.380570    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xb4zs_1d932d2e-0c6f-4068-8358-6d091cc9b123/dns-node-resolver/0.log"
Sep 16 13:43:33.613216 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:33.613175    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:33.613599 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:33.613296    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:33.613599 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:33.613319    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:43:33.613599 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:33.613402    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.613380841 +0000 UTC m=+38.240862200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : configmap "kube-rbac-proxy" not found
Sep 16 13:43:33.613599 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:33.613418    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:33.613599 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:33.613476    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:35.613459848 +0000 UTC m=+38.240941208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:34.017573 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.017535    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:34.017742 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.017596    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:34.017742 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:34.017691    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 13:43:34.017742 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:34.017712    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7446ffdbbc-qwcxh: secret "image-registry-tls" not found
Sep 16 13:43:34.017742 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:34.017735    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:43:34.017962 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:34.017777    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls podName:51726060-7491-449a-9591-9cc5c8bab5b1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:38.017758952 +0000 UTC m=+40.645240309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls") pod "image-registry-7446ffdbbc-qwcxh" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1") : secret "image-registry-tls" not found
Sep 16 13:43:34.017962 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:34.017794    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls podName:ef388d95-2cfe-4771-aa31-0128b60b0723 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:38.017785551 +0000 UTC m=+40.645266903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls") pod "dns-default-bxzwx" (UID: "ef388d95-2cfe-4771-aa31-0128b60b0723") : secret "dns-default-metrics-tls" not found
Sep 16 13:43:34.523987 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.523957    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-l84j4"]
Sep 16 13:43:34.526580 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.526555    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.529616 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.529462    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 13:43:34.529616 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.529518    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 13:43:34.529616 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.529580    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 13:43:34.529854 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.529524    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 13:43:34.530668 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.530647    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wddhx\""
Sep 16 13:43:34.546799 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.546778    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-l84j4"]
Sep 16 13:43:34.565897 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.565865    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz"]
Sep 16 13:43:34.570524 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:34.570496    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa72ef3_33ed_486b_b91f_648f4be57a47.slice/crio-501b5e5f791d4fb2f42fb195ad28e14a4b2ca27a2bb23e9d8c59b9e2972742bd WatchSource:0}: Error finding container 501b5e5f791d4fb2f42fb195ad28e14a4b2ca27a2bb23e9d8c59b9e2972742bd: Status 404 returned error can't find the container with id 501b5e5f791d4fb2f42fb195ad28e14a4b2ca27a2bb23e9d8c59b9e2972742bd
Sep 16 13:43:34.588879 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.588863    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sklmz_809e016e-5879-42bd-83c4-ea90a7d66cfd/node-ca/0.log"
Sep 16 13:43:34.623040 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.623017    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/163eaebe-06df-40ed-b761-76d661b71594-signing-key\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.623494 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.623045    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pw2\" (UniqueName: \"kubernetes.io/projected/163eaebe-06df-40ed-b761-76d661b71594-kube-api-access-p7pw2\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.623494 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.623103    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/163eaebe-06df-40ed-b761-76d661b71594-signing-cabundle\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.724459 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.724429    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/163eaebe-06df-40ed-b761-76d661b71594-signing-key\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.724459 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.724463    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pw2\" (UniqueName: \"kubernetes.io/projected/163eaebe-06df-40ed-b761-76d661b71594-kube-api-access-p7pw2\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.724661 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.724522    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/163eaebe-06df-40ed-b761-76d661b71594-signing-cabundle\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.725109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.725088    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/163eaebe-06df-40ed-b761-76d661b71594-signing-cabundle\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.728257 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.728221    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/163eaebe-06df-40ed-b761-76d661b71594-signing-key\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.737082 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.737053    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pw2\" (UniqueName: \"kubernetes.io/projected/163eaebe-06df-40ed-b761-76d661b71594-kube-api-access-p7pw2\") pod \"service-ca-568869fdc5-l84j4\" (UID: \"163eaebe-06df-40ed-b761-76d661b71594\") " pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:34.836627 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:34.836570    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-l84j4"
Sep 16 13:43:35.114571 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.114387    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-l84j4"]
Sep 16 13:43:35.118771 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:35.118742    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163eaebe_06df_40ed_b761_76d661b71594.slice/crio-633fa40a27d6c27fea88269f5c086169ae35711275e75286e4124d347d2b0567 WatchSource:0}: Error finding container 633fa40a27d6c27fea88269f5c086169ae35711275e75286e4124d347d2b0567: Status 404 returned error can't find the container with id 633fa40a27d6c27fea88269f5c086169ae35711275e75286e4124d347d2b0567
Sep 16 13:43:35.158415 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.158382    2548 generic.go:356] "Generic (PLEG): container finished" podID="944bfb3d-63e7-449f-9640-ffb1b5ae1a6a" containerID="bfbeb782c0920b9422a0bde9158fcdc2639fe5ba758979e4d67615456ddcb7b3" exitCode=0
Sep 16 13:43:35.158583 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.158466    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g775r" event={"ID":"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a","Type":"ContainerDied","Data":"bfbeb782c0920b9422a0bde9158fcdc2639fe5ba758979e4d67615456ddcb7b3"}
Sep 16 13:43:35.160151 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.160125    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gh2dv" event={"ID":"5cca1368-1fe3-483a-a299-13b6f79aa8ac","Type":"ContainerStarted","Data":"66cdfb0a4563f0f3b329ce61a1716b9dbaa752fbd71c6110c66aaba6b6fc65c7"}
Sep 16 13:43:35.161374 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.161347    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-l84j4" event={"ID":"163eaebe-06df-40ed-b761-76d661b71594","Type":"ContainerStarted","Data":"633fa40a27d6c27fea88269f5c086169ae35711275e75286e4124d347d2b0567"}
Sep 16 13:43:35.162543 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.162523    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz" event={"ID":"0aa72ef3-33ed-486b-b91f-648f4be57a47","Type":"ContainerStarted","Data":"501b5e5f791d4fb2f42fb195ad28e14a4b2ca27a2bb23e9d8c59b9e2972742bd"}
Sep 16 13:43:35.191381 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.190657    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gh2dv_5cca1368-1fe3-483a-a299-13b6f79aa8ac/serve-healthcheck-canary/0.log"
Sep 16 13:43:35.215304 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.215261    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gh2dv" podStartSLOduration=1.817657508 podStartE2EDuration="5.215228971s" totalImagesPullingTime="3.397571463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:35.215055718 +0000 UTC m=+37.842537123" watchObservedRunningTime="2026-09-16 13:43:35.215228971 +0000 UTC m=+37.842710350"
Sep 16 13:43:35.633090 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.633053    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:35.633949 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:35.633130    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:35.633949 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:35.633217    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:35.633949 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:35.633269    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:43:35.633949 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:35.633290    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:39.633271056 +0000 UTC m=+42.260752427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:35.633949 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:35.633317    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:39.633303705 +0000 UTC m=+42.260785056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : configmap "kube-rbac-proxy" not found
Sep 16 13:43:36.168505 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:36.168469    2548 generic.go:356] "Generic (PLEG): container finished" podID="944bfb3d-63e7-449f-9640-ffb1b5ae1a6a" containerID="eb18647a753f2cc1333261810db7df39ac433511e51d8f2ec9a25a5c44a8cb71" exitCode=0
Sep 16 13:43:36.168672 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:36.168552    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g775r" event={"ID":"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a","Type":"ContainerDied","Data":"eb18647a753f2cc1333261810db7df39ac433511e51d8f2ec9a25a5c44a8cb71"}
Sep 16 13:43:36.170290 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:36.170268    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz" event={"ID":"0aa72ef3-33ed-486b-b91f-648f4be57a47","Type":"ContainerStarted","Data":"23cddce2fd742b53cb9875f07c6e74f09a01e71c8db575b67a06866ca3112fa3"}
Sep 16 13:43:36.170387 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:36.170309    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz" event={"ID":"0aa72ef3-33ed-486b-b91f-648f4be57a47","Type":"ContainerStarted","Data":"2a566770a7a3b04185ded994e5b93d45cf9a3685ab9998ebcdada9f1ebdbf962"}
Sep 16 13:43:36.225299 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:36.225200    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ll8lz" podStartSLOduration=3.309392652 podStartE2EDuration="4.225181275s" totalImagesPullingTime="915.788623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:36.225061944 +0000 UTC m=+38.852543319" watchObservedRunningTime="2026-09-16 13:43:36.225181275 +0000 UTC m=+38.852662666"
Sep 16 13:43:37.175320 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:37.175280    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g775r" event={"ID":"944bfb3d-63e7-449f-9640-ffb1b5ae1a6a","Type":"ContainerStarted","Data":"7808b93c105e4a19a4ee2124264ca451cba1c20bf8a5c60f122df85d88421a48"}
Sep 16 13:43:37.207222 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:37.207092    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g775r" podStartSLOduration=13.255605415 podStartE2EDuration="40.207074679s" totalImagesPullingTime="26.951469264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:37.205735345 +0000 UTC m=+39.833216718" watchObservedRunningTime="2026-09-16 13:43:37.207074679 +0000 UTC m=+39.834556050"
Sep 16 13:43:38.052816 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:38.052781    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:38.052996 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:38.052826    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:38.052996 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:38.052923    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 13:43:38.052996 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:38.052944    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7446ffdbbc-qwcxh: secret "image-registry-tls" not found
Sep 16 13:43:38.052996 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:38.052933    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:43:38.052996 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:38.052992    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls podName:51726060-7491-449a-9591-9cc5c8bab5b1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:46.05297826 +0000 UTC m=+48.680459612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls") pod "image-registry-7446ffdbbc-qwcxh" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1") : secret "image-registry-tls" not found
Sep 16 13:43:38.053271 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:38.053008    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls podName:ef388d95-2cfe-4771-aa31-0128b60b0723 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:46.052998181 +0000 UTC m=+48.680479533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls") pod "dns-default-bxzwx" (UID: "ef388d95-2cfe-4771-aa31-0128b60b0723") : secret "dns-default-metrics-tls" not found
Sep 16 13:43:38.179344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:38.179308    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-l84j4" event={"ID":"163eaebe-06df-40ed-b761-76d661b71594","Type":"ContainerStarted","Data":"d6d0f3f49418685e809784ed55f56a94a785b54559c30409ea737cc539c16a30"}
Sep 16 13:43:38.202892 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:38.202844    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-l84j4" podStartSLOduration=1.878028072 podStartE2EDuration="4.202828889s" totalImagesPullingTime="2.324800817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:38.201853132 +0000 UTC m=+40.829334510" watchObservedRunningTime="2026-09-16 13:43:38.202828889 +0000 UTC m=+40.830310263"
Sep 16 13:43:39.665758 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:39.665721    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:39.666162 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:39.665779    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:39.666162 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:39.665881    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:39.666162 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:43:39.665958    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls podName:d1ba3d15-8156-4d5d-916f-e6538b6b91a1 nodeName:}" failed. No retries permitted until 2026-09-16 13:43:47.665936496 +0000 UTC m=+50.293417854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shkw6" (UID: "d1ba3d15-8156-4d5d-916f-e6538b6b91a1") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:43:39.666313 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:39.666297    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:46.112260 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:46.112210    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:46.112637 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:46.112281    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:46.115024 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:46.115001    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef388d95-2cfe-4771-aa31-0128b60b0723-metrics-tls\") pod \"dns-default-bxzwx\" (UID: \"ef388d95-2cfe-4771-aa31-0128b60b0723\") " pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:46.115197 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:46.115179    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"image-registry-7446ffdbbc-qwcxh\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") " pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:46.166082 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:46.166056    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:46.185273 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:46.185240    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:46.477109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:46.477078    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7446ffdbbc-qwcxh"]
Sep 16 13:43:46.480119 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:46.480092    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51726060_7491_449a_9591_9cc5c8bab5b1.slice/crio-147324da5e97b972c863409d50ea8d43f14c95a83686c1b02169c3bc7b9802c5 WatchSource:0}: Error finding container 147324da5e97b972c863409d50ea8d43f14c95a83686c1b02169c3bc7b9802c5: Status 404 returned error can't find the container with id 147324da5e97b972c863409d50ea8d43f14c95a83686c1b02169c3bc7b9802c5
Sep 16 13:43:46.505448 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:46.505426    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bxzwx"]
Sep 16 13:43:46.509241 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:46.509217    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef388d95_2cfe_4771_aa31_0128b60b0723.slice/crio-36b812de1771e8d830b779d61fe7e297e6236814c2afc4b01616441a7d958158 WatchSource:0}: Error finding container 36b812de1771e8d830b779d61fe7e297e6236814c2afc4b01616441a7d958158: Status 404 returned error can't find the container with id 36b812de1771e8d830b779d61fe7e297e6236814c2afc4b01616441a7d958158
Sep 16 13:43:47.205846 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:47.205785    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh" event={"ID":"51726060-7491-449a-9591-9cc5c8bab5b1","Type":"ContainerStarted","Data":"aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb"}
Sep 16 13:43:47.206340 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:47.205867    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh" event={"ID":"51726060-7491-449a-9591-9cc5c8bab5b1","Type":"ContainerStarted","Data":"147324da5e97b972c863409d50ea8d43f14c95a83686c1b02169c3bc7b9802c5"}
Sep 16 13:43:47.206340 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:47.205899    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:43:47.206980 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:47.206955    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bxzwx" event={"ID":"ef388d95-2cfe-4771-aa31-0128b60b0723","Type":"ContainerStarted","Data":"36b812de1771e8d830b779d61fe7e297e6236814c2afc4b01616441a7d958158"}
Sep 16 13:43:47.240835 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:47.240792    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh" podStartSLOduration=43.240773204 podStartE2EDuration="43.240773204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:47.240755765 +0000 UTC m=+49.868237142" watchObservedRunningTime="2026-09-16 13:43:47.240773204 +0000 UTC m=+49.868254579"
Sep 16 13:43:47.727976 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:47.727938    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:47.730439 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:47.730417    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1ba3d15-8156-4d5d-916f-e6538b6b91a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shkw6\" (UID: \"d1ba3d15-8156-4d5d-916f-e6538b6b91a1\") " pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:47.755355 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:47.755327    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-shkw6"
Sep 16 13:43:48.406222 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:48.406152    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-shkw6"]
Sep 16 13:43:48.489342 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:48.489300    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ba3d15_8156_4d5d_916f_e6538b6b91a1.slice/crio-fca12747f6bce1f6c6ee7bf42f24796a1f26996e70c841885ab205038122f076 WatchSource:0}: Error finding container fca12747f6bce1f6c6ee7bf42f24796a1f26996e70c841885ab205038122f076: Status 404 returned error can't find the container with id fca12747f6bce1f6c6ee7bf42f24796a1f26996e70c841885ab205038122f076
Sep 16 13:43:49.217612 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:49.217574    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bxzwx" event={"ID":"ef388d95-2cfe-4771-aa31-0128b60b0723","Type":"ContainerStarted","Data":"85a640b1fefe89c502aa03fa4f1a52bcfae69f44b1b77ee54cd47a8fe8a8ca0c"}
Sep 16 13:43:49.217788 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:49.217626    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bxzwx" event={"ID":"ef388d95-2cfe-4771-aa31-0128b60b0723","Type":"ContainerStarted","Data":"3bcb9e0580acbe588cb0bcdf104601c9c35878c3414733e2329941a66fde529b"}
Sep 16 13:43:49.217788 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:49.217665    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:43:49.219159 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:49.219137    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shkw6" event={"ID":"d1ba3d15-8156-4d5d-916f-e6538b6b91a1","Type":"ContainerStarted","Data":"3bbb41bf62531c0b2bfd30e2d23d82667282d544eac6813bfce1cb00d2ad345e"}
Sep 16 13:43:49.219282 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:49.219175    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shkw6" event={"ID":"d1ba3d15-8156-4d5d-916f-e6538b6b91a1","Type":"ContainerStarted","Data":"fca12747f6bce1f6c6ee7bf42f24796a1f26996e70c841885ab205038122f076"}
Sep 16 13:43:50.223903 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:50.223868    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shkw6" event={"ID":"d1ba3d15-8156-4d5d-916f-e6538b6b91a1","Type":"ContainerStarted","Data":"fe83e553fcc6c65c823ec26401a2e54dc571f765618c633a75a812bdb6479ee8"}
Sep 16 13:43:51.229553 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:51.229518    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shkw6" event={"ID":"d1ba3d15-8156-4d5d-916f-e6538b6b91a1","Type":"ContainerStarted","Data":"05a81ae120318d011bef2e2bdeb7ef7fcea20a37e6ae58f2fb32225a55d93595"}
Sep 16 13:43:51.269421 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:51.269333    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bxzwx" podStartSLOduration=19.260107833 podStartE2EDuration="21.269319114s" totalImagesPullingTime="2.009211281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:49.248947119 +0000 UTC m=+51.876428504" watchObservedRunningTime="2026-09-16 13:43:51.269319114 +0000 UTC m=+53.896800487"
Sep 16 13:43:51.269634 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:51.269604    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-shkw6" podStartSLOduration=17.955623749 podStartE2EDuration="20.269598859s" totalImagesPullingTime="2.31397511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:43:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:51.266767772 +0000 UTC m=+53.894249146" watchObservedRunningTime="2026-09-16 13:43:51.269598859 +0000 UTC m=+53.897080233"
Sep 16 13:43:56.948336 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.948299    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"]
Sep 16 13:43:56.975203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.975173    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:56.981605 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.981521    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 13:43:56.982431 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.982414    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 13:43:56.983336 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.983321    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 13:43:56.985503 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.985487    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 13:43:56.985756 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.985742    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 13:43:56.986013 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.986000    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 13:43:56.986763 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.986747    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 13:43:56.997039 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.997012    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88tlw\" (UniqueName: \"kubernetes.io/projected/60dd7065-53d5-4b23-8a1a-fddf7e69b563-kube-api-access-88tlw\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:56.997121 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.997053    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:56.997121 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.997081    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-hub\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:56.997224 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.997139    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:56.997224 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.997167    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-ca\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:56.997224 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:56.997211    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60dd7065-53d5-4b23-8a1a-fddf7e69b563-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.005873 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.005853    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"]
Sep 16 13:43:57.097821 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.097788    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.098005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.097827    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-ca\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.098005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.097849    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60dd7065-53d5-4b23-8a1a-fddf7e69b563-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.098005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.097914    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88tlw\" (UniqueName: \"kubernetes.io/projected/60dd7065-53d5-4b23-8a1a-fddf7e69b563-kube-api-access-88tlw\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.098154 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.098004    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.098154 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.098037    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-hub\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.098876 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.098839    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60dd7065-53d5-4b23-8a1a-fddf7e69b563-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.100512 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.100483    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-ca\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.100512 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.100485    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.100746 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.100724    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-hub\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.100813 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.100797    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60dd7065-53d5-4b23-8a1a-fddf7e69b563-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.134808 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.134781    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88tlw\" (UniqueName: \"kubernetes.io/projected/60dd7065-53d5-4b23-8a1a-fddf7e69b563-kube-api-access-88tlw\") pod \"cluster-proxy-proxy-agent-85577c6c99-85mzg\" (UID: \"60dd7065-53d5-4b23-8a1a-fddf7e69b563\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.295998 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.295935    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"
Sep 16 13:43:57.592218 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:57.592148    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg"]
Sep 16 13:43:57.594715 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:43:57.594692    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60dd7065_53d5_4b23_8a1a_fddf7e69b563.slice/crio-443cbf1f90b79a86abe9c5e9b1e3aff90bd95fb856d7ea223aa343d38ed40f0c WatchSource:0}: Error finding container 443cbf1f90b79a86abe9c5e9b1e3aff90bd95fb856d7ea223aa343d38ed40f0c: Status 404 returned error can't find the container with id 443cbf1f90b79a86abe9c5e9b1e3aff90bd95fb856d7ea223aa343d38ed40f0c
Sep 16 13:43:58.141961 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:58.141935    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhf7x"
Sep 16 13:43:58.250867 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:58.250836    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg" event={"ID":"60dd7065-53d5-4b23-8a1a-fddf7e69b563","Type":"ContainerStarted","Data":"443cbf1f90b79a86abe9c5e9b1e3aff90bd95fb856d7ea223aa343d38ed40f0c"}
Sep 16 13:43:59.226180 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:43:59.226144    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bxzwx"
Sep 16 13:44:02.270092 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:02.270057    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg" event={"ID":"60dd7065-53d5-4b23-8a1a-fddf7e69b563","Type":"ContainerStarted","Data":"7fd996ea83de88215746a98a31bd1fade636e58a7b5267a737ae8f135f9449c0"}
Sep 16 13:44:02.270583 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:02.270102    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg" event={"ID":"60dd7065-53d5-4b23-8a1a-fddf7e69b563","Type":"ContainerStarted","Data":"8cb2c1b7ce823dadc2a4919b46b336f2d67826835067dfb5a79580231202aefa"}
Sep 16 13:44:02.270583 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:02.270119    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg" event={"ID":"60dd7065-53d5-4b23-8a1a-fddf7e69b563","Type":"ContainerStarted","Data":"6a0a284e293de199207f3c621fd5176cd4557b0d2f5e3d627e82ff9caa34f116"}
Sep 16 13:44:02.301461 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:02.301414    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85577c6c99-85mzg" podStartSLOduration=2.3860097639999998 podStartE2EDuration="6.301398567s" totalImagesPullingTime="3.915388803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:02.299380992 +0000 UTC m=+64.926862367" watchObservedRunningTime="2026-09-16 13:44:02.301398567 +0000 UTC m=+64.928879941"
Sep 16 13:44:03.647142 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.647106    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:44:03.647142 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.647144    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:44:03.650130 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.650101    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 13:44:03.651001 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.650985    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 13:44:03.660367 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.660342    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4f50b31-1baf-4e33-8e24-9826ee73ace4-metrics-certs\") pod \"network-metrics-daemon-hdb69\" (UID: \"a4f50b31-1baf-4e33-8e24-9826ee73ace4\") " pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:44:03.660466 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.660392    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdae81bc-d373-442e-90b2-a48654b28fb6-original-pull-secret\") pod \"global-pull-secret-syncer-wqkrw\" (UID: \"cdae81bc-d373-442e-90b2-a48654b28fb6\") " pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:44:03.747985 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.747949    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:44:03.756282 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.756242    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 13:44:03.760679 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.760663    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 13:44:03.771254 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.771231    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbvfp\" (UniqueName: \"kubernetes.io/projected/8e73d474-7109-4801-8634-a735f2dc3ad8-kube-api-access-lbvfp\") pod \"network-check-target-9kx98\" (UID: \"8e73d474-7109-4801-8634-a735f2dc3ad8\") " pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:44:03.777415 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.777395    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4knh2\""
Sep 16 13:44:03.782499 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.782478    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqkrw"
Sep 16 13:44:03.785600 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.785584    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:44:03.790047 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.790033    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7xrvw\""
Sep 16 13:44:03.797940 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:03.797926    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hdb69"
Sep 16 13:44:04.069532 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:04.069509    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wqkrw"]
Sep 16 13:44:04.071890 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:44:04.071868    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdae81bc_d373_442e_90b2_a48654b28fb6.slice/crio-75cfffe851fdad5c616d72c0e18d6ff15bf7fec2c11a23c483e8d548e8d68152 WatchSource:0}: Error finding container 75cfffe851fdad5c616d72c0e18d6ff15bf7fec2c11a23c483e8d548e8d68152: Status 404 returned error can't find the container with id 75cfffe851fdad5c616d72c0e18d6ff15bf7fec2c11a23c483e8d548e8d68152
Sep 16 13:44:04.097902 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:04.097733    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9kx98"]
Sep 16 13:44:04.098758 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:04.098673    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hdb69"]
Sep 16 13:44:04.100608 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:44:04.100588    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f50b31_1baf_4e33_8e24_9826ee73ace4.slice/crio-23d8b40ae1540c3decb4435758dd475d0516837faa26a5d7659ed3147b3ad379 WatchSource:0}: Error finding container 23d8b40ae1540c3decb4435758dd475d0516837faa26a5d7659ed3147b3ad379: Status 404 returned error can't find the container with id 23d8b40ae1540c3decb4435758dd475d0516837faa26a5d7659ed3147b3ad379
Sep 16 13:44:04.101210 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:44:04.101166    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e73d474_7109_4801_8634_a735f2dc3ad8.slice/crio-611adde337689d723b132c70885e4ea226e7bda54e17a102afa15151fefa1ec0 WatchSource:0}: Error finding container 611adde337689d723b132c70885e4ea226e7bda54e17a102afa15151fefa1ec0: Status 404 returned error can't find the container with id 611adde337689d723b132c70885e4ea226e7bda54e17a102afa15151fefa1ec0
Sep 16 13:44:04.277638 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:04.277604    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9kx98" event={"ID":"8e73d474-7109-4801-8634-a735f2dc3ad8","Type":"ContainerStarted","Data":"611adde337689d723b132c70885e4ea226e7bda54e17a102afa15151fefa1ec0"}
Sep 16 13:44:04.278760 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:04.278733    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wqkrw" event={"ID":"cdae81bc-d373-442e-90b2-a48654b28fb6","Type":"ContainerStarted","Data":"75cfffe851fdad5c616d72c0e18d6ff15bf7fec2c11a23c483e8d548e8d68152"}
Sep 16 13:44:04.279869 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:04.279847    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hdb69" event={"ID":"a4f50b31-1baf-4e33-8e24-9826ee73ace4","Type":"ContainerStarted","Data":"23d8b40ae1540c3decb4435758dd475d0516837faa26a5d7659ed3147b3ad379"}
Sep 16 13:44:06.292291 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:06.292180    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hdb69" event={"ID":"a4f50b31-1baf-4e33-8e24-9826ee73ace4","Type":"ContainerStarted","Data":"6c28ecf8687a3f76ddf32f606510f2b6755806ba20e788d97b1ad6e3ed4cb55b"}
Sep 16 13:44:06.292291 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:06.292269    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hdb69" event={"ID":"a4f50b31-1baf-4e33-8e24-9826ee73ace4","Type":"ContainerStarted","Data":"a539a231869f63857b54a3957067f6b5b64bf54cf4249886a747e7fb480e899f"}
Sep 16 13:44:08.216588 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.216553    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:44:08.260967 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.260914    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hdb69" podStartSLOduration=68.586496806 podStartE2EDuration="1m10.260897228s" totalImagesPullingTime="1.674400422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:06.322298553 +0000 UTC m=+68.949780056" watchObservedRunningTime="2026-09-16 13:44:08.260897228 +0000 UTC m=+70.888378602"
Sep 16 13:44:08.895529 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.895496    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9tt2w"]
Sep 16 13:44:08.898900 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.898875    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.901753 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.901627    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 13:44:08.901753 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.901724    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 13:44:08.902173 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.902006    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 13:44:08.902173 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.902096    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 13:44:08.902802 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.902785    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 13:44:08.902922 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.902888    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 13:44:08.903239 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.903124    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5m2g2\""
Sep 16 13:44:08.931669 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.931641    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"]
Sep 16 13:44:08.934152 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.934131    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:08.937565 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.937546    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 13:44:08.937565 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.937556    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rtczd\""
Sep 16 13:44:08.937769 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.937571    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 13:44:08.937894 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.937853    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 13:44:08.950939 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.950866    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"]
Sep 16 13:44:08.989864 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.989833    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:08.989991 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.989876    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:08.989991 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.989909    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:08.989991 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.989938    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.989991 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.989976    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-tls\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.990200 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990024    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-sys\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.990200 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990054    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-textfile\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.990200 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990113    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhkj4\" (UniqueName: \"kubernetes.io/projected/51b9a334-5b84-4317-a09d-06a3ece75cbe-kube-api-access-xhkj4\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.990200 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990145    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-wtmp\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.990200 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990173    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scr2\" (UniqueName: \"kubernetes.io/projected/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-api-access-5scr2\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:08.990462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990210    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b9a334-5b84-4317-a09d-06a3ece75cbe-metrics-client-ca\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.990462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990267    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:08.990462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990291    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-root\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:08.990462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990318    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:08.990462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:08.990343    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.091551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091515    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.091721 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091571    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-tls\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.091721 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091602    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-sys\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.091721 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091630    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-textfile\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.091721 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091667    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhkj4\" (UniqueName: \"kubernetes.io/projected/51b9a334-5b84-4317-a09d-06a3ece75cbe-kube-api-access-xhkj4\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.091721 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091700    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-wtmp\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091727    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scr2\" (UniqueName: \"kubernetes.io/projected/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-api-access-5scr2\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091765    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b9a334-5b84-4317-a09d-06a3ece75cbe-metrics-client-ca\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091795    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091820    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-root\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091855    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091882    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091910    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091934    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.092018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091967    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.092479 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091882    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-wtmp\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092479 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.092156    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-accelerators-collector-config\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092479 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.092329    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.092479 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.091728    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-sys\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092675 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.092654    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-textfile\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092747 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.092726    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.092806 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.092784    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b9a334-5b84-4317-a09d-06a3ece75cbe-root\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.092859 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.092830    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b9a334-5b84-4317-a09d-06a3ece75cbe-metrics-client-ca\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.093296 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.093216    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.095130 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.095107    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.095261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.095220    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b9a334-5b84-4317-a09d-06a3ece75cbe-node-exporter-tls\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.095366 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.095260    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.095366 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.095342    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.150881 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.150639    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhkj4\" (UniqueName: \"kubernetes.io/projected/51b9a334-5b84-4317-a09d-06a3ece75cbe-kube-api-access-xhkj4\") pod \"node-exporter-9tt2w\" (UID: \"51b9a334-5b84-4317-a09d-06a3ece75cbe\") " pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.150881 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.150725    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scr2\" (UniqueName: \"kubernetes.io/projected/7c6e42ac-e552-49bb-879d-1c37ad6cfb35-kube-api-access-5scr2\") pod \"kube-state-metrics-59fbdd77d-2v8pl\" (UID: \"7c6e42ac-e552-49bb-879d-1c37ad6cfb35\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.211875 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.211833    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9tt2w"
Sep 16 13:44:09.245857 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.245834    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"
Sep 16 13:44:09.257326 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:44:09.257299    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b9a334_5b84_4317_a09d_06a3ece75cbe.slice/crio-0600135a45ccd0ccbb3f0e18a6cf22e65cdc699e8e5071e86da0718a58fe9d3c WatchSource:0}: Error finding container 0600135a45ccd0ccbb3f0e18a6cf22e65cdc699e8e5071e86da0718a58fe9d3c: Status 404 returned error can't find the container with id 0600135a45ccd0ccbb3f0e18a6cf22e65cdc699e8e5071e86da0718a58fe9d3c
Sep 16 13:44:09.312470 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.312391    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tt2w" event={"ID":"51b9a334-5b84-4317-a09d-06a3ece75cbe","Type":"ContainerStarted","Data":"0600135a45ccd0ccbb3f0e18a6cf22e65cdc699e8e5071e86da0718a58fe9d3c"}
Sep 16 13:44:09.547220 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:09.547151    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl"]
Sep 16 13:44:09.550427 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:44:09.550397    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6e42ac_e552_49bb_879d_1c37ad6cfb35.slice/crio-a3b6c276e3e0a626ff154e19d18b4c083e942c6cd1e95a92f4a92fd4d01655de WatchSource:0}: Error finding container a3b6c276e3e0a626ff154e19d18b4c083e942c6cd1e95a92f4a92fd4d01655de: Status 404 returned error can't find the container with id a3b6c276e3e0a626ff154e19d18b4c083e942c6cd1e95a92f4a92fd4d01655de
Sep 16 13:44:10.318269 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:10.318220    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wqkrw" event={"ID":"cdae81bc-d373-442e-90b2-a48654b28fb6","Type":"ContainerStarted","Data":"ed01b0ad0095688c8f3092a0c2bbf1d1f1f91c695b6ece1bc9e738144006d447"}
Sep 16 13:44:10.320599 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:10.320535    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tt2w" event={"ID":"51b9a334-5b84-4317-a09d-06a3ece75cbe","Type":"ContainerStarted","Data":"fe98b91ca6fa66a6fc2eb7a0fcff5f80ebfdb7a6637f7945d02a78e1b00c610c"}
Sep 16 13:44:10.322547 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:10.322509    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9kx98" event={"ID":"8e73d474-7109-4801-8634-a735f2dc3ad8","Type":"ContainerStarted","Data":"922f1e8fefcf49aec74939a364c94ea065285676da11a1ae5e1cb033fdc61401"}
Sep 16 13:44:10.322668 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:10.322580    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:44:10.324013 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:10.323984    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl" event={"ID":"7c6e42ac-e552-49bb-879d-1c37ad6cfb35","Type":"ContainerStarted","Data":"a3b6c276e3e0a626ff154e19d18b4c083e942c6cd1e95a92f4a92fd4d01655de"}
Sep 16 13:44:10.344196 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:10.344161    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wqkrw" podStartSLOduration=67.154384997 podStartE2EDuration="1m12.344147814s" totalImagesPullingTime="5.189762817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:10.340887031 +0000 UTC m=+72.968368428" watchObservedRunningTime="2026-09-16 13:44:10.344147814 +0000 UTC m=+72.971629188"
Sep 16 13:44:10.417035 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:10.416984    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9kx98" podStartSLOduration=68.253961442 podStartE2EDuration="1m13.416964965s" totalImagesPullingTime="5.163003523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:10.415309383 +0000 UTC m=+73.042790759" watchObservedRunningTime="2026-09-16 13:44:10.416964965 +0000 UTC m=+73.044446338"
Sep 16 13:44:11.329492 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:11.329454    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl" event={"ID":"7c6e42ac-e552-49bb-879d-1c37ad6cfb35","Type":"ContainerStarted","Data":"a03f44a510c19323defc2329b2bd7e7edfcb8fc325f6fb9cb1b6760be3c7b885"}
Sep 16 13:44:11.329927 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:11.329513    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl" event={"ID":"7c6e42ac-e552-49bb-879d-1c37ad6cfb35","Type":"ContainerStarted","Data":"4e55b8c35cb9f3e1a36c282ecdc4b21297e90e414733a5460743cbfdadc9938c"}
Sep 16 13:44:11.329927 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:11.329535    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl" event={"ID":"7c6e42ac-e552-49bb-879d-1c37ad6cfb35","Type":"ContainerStarted","Data":"58a905ad0139b22d6271e6345ff9221220a23db00a392a6135571109291c3314"}
Sep 16 13:44:11.331032 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:11.331010    2548 generic.go:356] "Generic (PLEG): container finished" podID="51b9a334-5b84-4317-a09d-06a3ece75cbe" containerID="fe98b91ca6fa66a6fc2eb7a0fcff5f80ebfdb7a6637f7945d02a78e1b00c610c" exitCode=0
Sep 16 13:44:11.331116 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:11.331086    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tt2w" event={"ID":"51b9a334-5b84-4317-a09d-06a3ece75cbe","Type":"ContainerDied","Data":"fe98b91ca6fa66a6fc2eb7a0fcff5f80ebfdb7a6637f7945d02a78e1b00c610c"}
Sep 16 13:44:11.359880 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:11.359829    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2v8pl" podStartSLOduration=2.296495662 podStartE2EDuration="3.359812841s" totalImagesPullingTime="1.063317179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:11.359635587 +0000 UTC m=+73.987116960" watchObservedRunningTime="2026-09-16 13:44:11.359812841 +0000 UTC m=+73.987294228"
Sep 16 13:44:12.336576 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:12.336539    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tt2w" event={"ID":"51b9a334-5b84-4317-a09d-06a3ece75cbe","Type":"ContainerStarted","Data":"55c4f31c78a3d60db085a536064dc9cfb34d8047047a937c73fcbe10089f84a9"}
Sep 16 13:44:12.337005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:12.336608    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9tt2w" event={"ID":"51b9a334-5b84-4317-a09d-06a3ece75cbe","Type":"ContainerStarted","Data":"e55a817bdb66a6746fd0b8927da189878b8d6e5cefc53d67f92ed1f5e8d238ca"}
Sep 16 13:44:12.374452 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:12.374403    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9tt2w" podStartSLOduration=3.377364689 podStartE2EDuration="4.374389905s" totalImagesPullingTime="928.817934ms" totalInitContainerRuntime="68.207282ms" isStatefulPod=true podCreationTimestamp="2026-09-16 13:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:12.37303046 +0000 UTC m=+75.000511860" watchObservedRunningTime="2026-09-16 13:44:12.374389905 +0000 UTC m=+75.001871279"
Sep 16 13:44:15.613408 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.613372    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:44:15.616029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.616005    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.622109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.622080    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-b78wk\""
Sep 16 13:44:15.622242 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.622158    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 13:44:15.622632 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.622082    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 13:44:15.622632 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.622398    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-48kr7e4rae9ba\""
Sep 16 13:44:15.622971 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.622952    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 13:44:15.623265 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.623223    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 13:44:15.623366 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.623227    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 13:44:15.623429 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.623376    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 13:44:15.624209 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.624195    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 13:44:15.624454 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.624433    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 13:44:15.624595 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.624573    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 13:44:15.624673 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.624575    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 13:44:15.625438 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.625419    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 13:44:15.625541 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.625445    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 13:44:15.625541 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.625532    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 13:44:15.626937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.626918    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 13:44:15.634690 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.631230    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 13:44:15.642510 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.642483    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:44:15.743885 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.743847    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.743885 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.743886    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744065 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.743914    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744065 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.743946    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-web-config\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744065 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744000    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744065 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744017    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744065 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744035    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744065 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744050    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744264 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744084    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744264 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744103    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744264 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744119    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-config-out\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744264 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744140    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-config\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744264 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744156    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744264 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744193    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpcs\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-kube-api-access-6cpcs\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744267    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744299    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744326    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744347    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744367    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.744436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.744423    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845024 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.844995    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845166 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845027    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845166 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845048    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845166 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845064    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-web-config\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845166 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845082    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845166 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845097    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845166 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845114    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845274    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845320    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845346    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845383    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-config-out\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845422    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-config\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845462 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845448    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845474    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpcs\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-kube-api-access-6cpcs\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845500    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845533    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845565    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845595    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845629    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.845740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845671    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.846063 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.845833    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.846152 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.846124    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.846298 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.846170    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.847138 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.846834    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.847138 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.847010    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.847363 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.847143    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.849945 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.848402    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.849945 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.848682    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.849945 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.848746    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.849945 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.848903    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.849945 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.849135    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-web-config\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.849945 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.849813    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.849945 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.849907    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.850396 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.850187    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-config-out\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.850396 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.850330    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.850595 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.850570    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.851080 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.851059    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-config\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.851265 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.851228    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.851476 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.851461    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.864046 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.864000    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpcs\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-kube-api-access-6cpcs\") pod \"prometheus-k8s-0\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:15.925919 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:15.925895    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:16.231298 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:16.231265    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:44:16.234035 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:44:16.234010    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789c7c94_9d42_4082_9003_d160aa4a2ec9.slice/crio-04d260d7f87d31aaf7dbfa36cca4bd7c17be6591993654c795a5a1a0a66c9589 WatchSource:0}: Error finding container 04d260d7f87d31aaf7dbfa36cca4bd7c17be6591993654c795a5a1a0a66c9589: Status 404 returned error can't find the container with id 04d260d7f87d31aaf7dbfa36cca4bd7c17be6591993654c795a5a1a0a66c9589
Sep 16 13:44:16.352723 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:16.352689    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerStarted","Data":"04d260d7f87d31aaf7dbfa36cca4bd7c17be6591993654c795a5a1a0a66c9589"}
Sep 16 13:44:18.360718 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:18.360686    2548 generic.go:356] "Generic (PLEG): container finished" podID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerID="4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867" exitCode=0
Sep 16 13:44:18.361073 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:18.360748    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerDied","Data":"4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867"}
Sep 16 13:44:19.800371 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:19.800334    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7446ffdbbc-qwcxh"]
Sep 16 13:44:21.376516 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:21.376101    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerStarted","Data":"cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27"}
Sep 16 13:44:21.376516 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:21.376188    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerStarted","Data":"5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847"}
Sep 16 13:44:24.393908 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:24.393867    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerStarted","Data":"197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5"}
Sep 16 13:44:24.394310 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:24.393936    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerStarted","Data":"ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743"}
Sep 16 13:44:24.394310 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:24.393964    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerStarted","Data":"8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776"}
Sep 16 13:44:24.394310 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:24.393993    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerStarted","Data":"907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb"}
Sep 16 13:44:24.433227 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:24.433177    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.493903654 podStartE2EDuration="9.433161071s" totalImagesPullingTime="5.920719098s" totalInitContainerRuntime="18.538319ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:24.430995544 +0000 UTC m=+87.058476941" watchObservedRunningTime="2026-09-16 13:44:24.433161071 +0000 UTC m=+87.060642445"
Sep 16 13:44:25.926221 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:25.926185    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:44:41.334099 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:41.334069    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9kx98"
Sep 16 13:44:44.822156 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:44.822089    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh" podUID="51726060-7491-449a-9591-9cc5c8bab5b1" containerName="registry" containerID="cri-o://aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb" gracePeriod=30
Sep 16 13:44:45.089040 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.088979    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:44:45.269335 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269308    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-registry-certificates\") pod \"51726060-7491-449a-9591-9cc5c8bab5b1\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") "
Sep 16 13:44:45.269493 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269342    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-trusted-ca\") pod \"51726060-7491-449a-9591-9cc5c8bab5b1\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") "
Sep 16 13:44:45.269493 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269380    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-image-registry-private-configuration\") pod \"51726060-7491-449a-9591-9cc5c8bab5b1\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") "
Sep 16 13:44:45.269493 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269423    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-bound-sa-token\") pod \"51726060-7491-449a-9591-9cc5c8bab5b1\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") "
Sep 16 13:44:45.269493 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269480    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51726060-7491-449a-9591-9cc5c8bab5b1-ca-trust-extracted\") pod \"51726060-7491-449a-9591-9cc5c8bab5b1\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") "
Sep 16 13:44:45.269703 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269503    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-installation-pull-secrets\") pod \"51726060-7491-449a-9591-9cc5c8bab5b1\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") "
Sep 16 13:44:45.269703 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269526    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gtrb\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-kube-api-access-5gtrb\") pod \"51726060-7491-449a-9591-9cc5c8bab5b1\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") "
Sep 16 13:44:45.269703 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269554    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") pod \"51726060-7491-449a-9591-9cc5c8bab5b1\" (UID: \"51726060-7491-449a-9591-9cc5c8bab5b1\") "
Sep 16 13:44:45.269844 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.269755    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "51726060-7491-449a-9591-9cc5c8bab5b1" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:44:45.272181 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.270464    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "51726060-7491-449a-9591-9cc5c8bab5b1" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:44:45.275871 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.275838    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "51726060-7491-449a-9591-9cc5c8bab5b1" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:44:45.277300 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.277225    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "51726060-7491-449a-9591-9cc5c8bab5b1" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:44:45.277300 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.277225    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-kube-api-access-5gtrb" (OuterVolumeSpecName: "kube-api-access-5gtrb") pod "51726060-7491-449a-9591-9cc5c8bab5b1" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1"). InnerVolumeSpecName "kube-api-access-5gtrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:44:45.277471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.277393    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "51726060-7491-449a-9591-9cc5c8bab5b1" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:44:45.277471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.277460    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "51726060-7491-449a-9591-9cc5c8bab5b1" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:44:45.280878 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.280853    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51726060-7491-449a-9591-9cc5c8bab5b1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "51726060-7491-449a-9591-9cc5c8bab5b1" (UID: "51726060-7491-449a-9591-9cc5c8bab5b1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:44:45.370859 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.370841    2548 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-registry-certificates\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:44:45.370936 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.370878    2548 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51726060-7491-449a-9591-9cc5c8bab5b1-trusted-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:44:45.370936 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.370888    2548 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-image-registry-private-configuration\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:44:45.370936 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.370897    2548 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-bound-sa-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:44:45.370936 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.370906    2548 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51726060-7491-449a-9591-9cc5c8bab5b1-ca-trust-extracted\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:44:45.370936 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.370914    2548 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51726060-7491-449a-9591-9cc5c8bab5b1-installation-pull-secrets\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:44:45.370936 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.370921    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gtrb\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-kube-api-access-5gtrb\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:44:45.370936 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.370929    2548 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51726060-7491-449a-9591-9cc5c8bab5b1-registry-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:44:45.464994 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.464965    2548 generic.go:356] "Generic (PLEG): container finished" podID="51726060-7491-449a-9591-9cc5c8bab5b1" containerID="aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb" exitCode=0
Sep 16 13:44:45.465121 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.465022    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh"
Sep 16 13:44:45.465121 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.465051    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh" event={"ID":"51726060-7491-449a-9591-9cc5c8bab5b1","Type":"ContainerDied","Data":"aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb"}
Sep 16 13:44:45.465234 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.465134    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7446ffdbbc-qwcxh" event={"ID":"51726060-7491-449a-9591-9cc5c8bab5b1","Type":"ContainerDied","Data":"147324da5e97b972c863409d50ea8d43f14c95a83686c1b02169c3bc7b9802c5"}
Sep 16 13:44:45.465234 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.465164    2548 scope.go:117] "RemoveContainer" containerID="aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb"
Sep 16 13:44:45.473281 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.473261    2548 scope.go:117] "RemoveContainer" containerID="aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb"
Sep 16 13:44:45.473554 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:44:45.473532    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb\": container with ID starting with aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb not found: ID does not exist" containerID="aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb"
Sep 16 13:44:45.473630 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.473564    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb"} err="failed to get container status \"aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb\": rpc error: code = NotFound desc = could not find container \"aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb\": container with ID starting with aba719c909f965231928a5e7ff9455b07d7236b34cd210ee7353a7debeb503fb not found: ID does not exist"
Sep 16 13:44:45.489814 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.489789    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7446ffdbbc-qwcxh"]
Sep 16 13:44:45.495665 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.495646    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7446ffdbbc-qwcxh"]
Sep 16 13:44:45.968386 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:44:45.968355    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51726060-7491-449a-9591-9cc5c8bab5b1" path="/var/lib/kubelet/pods/51726060-7491-449a-9591-9cc5c8bab5b1/volumes"
Sep 16 13:45:15.926200 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:15.926161    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:15.944921 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:15.944895    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:16.600998 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:16.600971    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:34.096041 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.096005    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:45:34.096551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.096336    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="prometheus" containerID="cri-o://5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847" gracePeriod=600
Sep 16 13:45:34.096551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.096352    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="config-reloader" containerID="cri-o://cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27" gracePeriod=600
Sep 16 13:45:34.096551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.096361    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743" gracePeriod=600
Sep 16 13:45:34.096551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.096347    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy-web" containerID="cri-o://907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb" gracePeriod=600
Sep 16 13:45:34.096551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.096365    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy" containerID="cri-o://8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776" gracePeriod=600
Sep 16 13:45:34.096551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.096440    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="thanos-sidecar" containerID="cri-o://197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5" gracePeriod=600
Sep 16 13:45:34.655694 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655663    2548 generic.go:356] "Generic (PLEG): container finished" podID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerID="ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743" exitCode=0
Sep 16 13:45:34.655694 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655688    2548 generic.go:356] "Generic (PLEG): container finished" podID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerID="8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776" exitCode=0
Sep 16 13:45:34.655694 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655696    2548 generic.go:356] "Generic (PLEG): container finished" podID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerID="197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5" exitCode=0
Sep 16 13:45:34.655694 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655709    2548 generic.go:356] "Generic (PLEG): container finished" podID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerID="cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27" exitCode=0
Sep 16 13:45:34.656025 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655714    2548 generic.go:356] "Generic (PLEG): container finished" podID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerID="5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847" exitCode=0
Sep 16 13:45:34.656025 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655735    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerDied","Data":"ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743"}
Sep 16 13:45:34.656025 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655814    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerDied","Data":"8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776"}
Sep 16 13:45:34.656025 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655854    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerDied","Data":"197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5"}
Sep 16 13:45:34.656025 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655890    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerDied","Data":"cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27"}
Sep 16 13:45:34.656025 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:34.655930    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerDied","Data":"5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847"}
Sep 16 13:45:35.338059 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.338033    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.427485 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427419    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-thanos-prometheus-http-client-file\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427485 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427454    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-metrics-client-certs\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427485 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427470    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-kube-rbac-proxy\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427485 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427490    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-db\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427515    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cpcs\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-kube-api-access-6cpcs\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427552    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-grpc-tls\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427582    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-serving-certs-ca-bundle\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427608    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-1\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427643    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-tls\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427667    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-config\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427692    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-2\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427726    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-0\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.427794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427755    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-trusted-ca-bundle\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.428241 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427816    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-config-out\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.428241 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427853    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-metrics-client-ca\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.428241 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427879    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.428241 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427912    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-web-config\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.428241 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427949    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-tls-assets\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.428241 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.427978    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.428241 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.428020    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-kubelet-serving-ca-bundle\") pod \"789c7c94-9d42-4082-9003-d160aa4a2ec9\" (UID: \"789c7c94-9d42-4082-9003-d160aa4a2ec9\") "
Sep 16 13:45:35.428676 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.428585    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:35.430079 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.428957    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:35.430079 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.429002    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:35.430079 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.429198    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:35.430079 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.429538    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:35.430079 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.429584    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:35.430079 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.429831    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:35.430719 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.430665    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:45:35.431353 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.431325    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.432076 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.432039    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-config" (OuterVolumeSpecName: "config") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.432169 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.432070    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.432477 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.432450    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.432571 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.432521    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-config-out" (OuterVolumeSpecName: "config-out") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:35.432844 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.432813    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-kube-api-access-6cpcs" (OuterVolumeSpecName: "kube-api-access-6cpcs") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "kube-api-access-6cpcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:35.433003 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.432975    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.433142 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.433079    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.433590 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.433512    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:35.433748 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.433661    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.433880 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.433856    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.443698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.443678    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-web-config" (OuterVolumeSpecName: "web-config") pod "789c7c94-9d42-4082-9003-d160aa4a2ec9" (UID: "789c7c94-9d42-4082-9003-d160aa4a2ec9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:35.529167 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529141    2548 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-metrics-client-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529167 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529164    2548 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-kube-rbac-proxy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529174    2548 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-db\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529182    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpcs\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-kube-api-access-6cpcs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529193    2548 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-grpc-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529203    2548 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529211    2548 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529220    2548 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529228    2548 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-config\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529236    2548 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529263    2548 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529272    2548 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529280    2548 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/789c7c94-9d42-4082-9003-d160aa4a2ec9-config-out\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529290    2548 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-metrics-client-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529300    2548 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529308    2548 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-web-config\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529317    2548 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/789c7c94-9d42-4082-9003-d160aa4a2ec9-tls-assets\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529325    2548 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529333    2548 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/789c7c94-9d42-4082-9003-d160aa4a2ec9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.529807 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.529342    2548 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/789c7c94-9d42-4082-9003-d160aa4a2ec9-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:45:35.662839 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.662812    2548 generic.go:356] "Generic (PLEG): container finished" podID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerID="907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb" exitCode=0
Sep 16 13:45:35.662937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.662846    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerDied","Data":"907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb"}
Sep 16 13:45:35.662937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.662887    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.662937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.662901    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"789c7c94-9d42-4082-9003-d160aa4a2ec9","Type":"ContainerDied","Data":"04d260d7f87d31aaf7dbfa36cca4bd7c17be6591993654c795a5a1a0a66c9589"}
Sep 16 13:45:35.662937 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.662934    2548 scope.go:117] "RemoveContainer" containerID="ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743"
Sep 16 13:45:35.670257 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.670207    2548 scope.go:117] "RemoveContainer" containerID="8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776"
Sep 16 13:45:35.677263 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.677231    2548 scope.go:117] "RemoveContainer" containerID="907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb"
Sep 16 13:45:35.683453 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.683431    2548 scope.go:117] "RemoveContainer" containerID="197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5"
Sep 16 13:45:35.689834 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.689771    2548 scope.go:117] "RemoveContainer" containerID="cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27"
Sep 16 13:45:35.692189 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.692169    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:45:35.696904 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.696889    2548 scope.go:117] "RemoveContainer" containerID="5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847"
Sep 16 13:45:35.698261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.698228    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:45:35.703562 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.703547    2548 scope.go:117] "RemoveContainer" containerID="4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867"
Sep 16 13:45:35.709596 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.709581    2548 scope.go:117] "RemoveContainer" containerID="ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743"
Sep 16 13:45:35.709864 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:45:35.709846    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743\": container with ID starting with ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743 not found: ID does not exist" containerID="ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743"
Sep 16 13:45:35.709908 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.709875    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743"} err="failed to get container status \"ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743\": rpc error: code = NotFound desc = could not find container \"ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743\": container with ID starting with ea3999fff18f033f966399fadf2042559622234d5c8e991d35ef30cf1f281743 not found: ID does not exist"
Sep 16 13:45:35.709908 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.709893    2548 scope.go:117] "RemoveContainer" containerID="8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776"
Sep 16 13:45:35.710146 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:45:35.710129    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776\": container with ID starting with 8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776 not found: ID does not exist" containerID="8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776"
Sep 16 13:45:35.710189 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.710152    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776"} err="failed to get container status \"8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776\": rpc error: code = NotFound desc = could not find container \"8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776\": container with ID starting with 8f36fe30844b59d706ef228075bf8ab26e85874be814242351754740dae8f776 not found: ID does not exist"
Sep 16 13:45:35.710189 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.710167    2548 scope.go:117] "RemoveContainer" containerID="907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb"
Sep 16 13:45:35.710435 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:45:35.710417    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb\": container with ID starting with 907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb not found: ID does not exist" containerID="907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb"
Sep 16 13:45:35.710499 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.710443    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb"} err="failed to get container status \"907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb\": rpc error: code = NotFound desc = could not find container \"907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb\": container with ID starting with 907a6afd58e1f26adfd67067beb3415cb2746cf33af4eda298750137ce1748eb not found: ID does not exist"
Sep 16 13:45:35.710499 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.710463    2548 scope.go:117] "RemoveContainer" containerID="197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5"
Sep 16 13:45:35.710737 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:45:35.710722    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5\": container with ID starting with 197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5 not found: ID does not exist" containerID="197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5"
Sep 16 13:45:35.710795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.710741    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5"} err="failed to get container status \"197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5\": rpc error: code = NotFound desc = could not find container \"197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5\": container with ID starting with 197b4b82e9aa62ba5240206a85238d410f53a1dffbf6737f195ac3373db597b5 not found: ID does not exist"
Sep 16 13:45:35.710795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.710759    2548 scope.go:117] "RemoveContainer" containerID="cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27"
Sep 16 13:45:35.711015 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:45:35.710998    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27\": container with ID starting with cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27 not found: ID does not exist" containerID="cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27"
Sep 16 13:45:35.711054 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.711019    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27"} err="failed to get container status \"cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27\": rpc error: code = NotFound desc = could not find container \"cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27\": container with ID starting with cd47a3336edac501fcf158fe44b690c33e9e667ff2abfd0d63d8adabd061ad27 not found: ID does not exist"
Sep 16 13:45:35.711054 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.711033    2548 scope.go:117] "RemoveContainer" containerID="5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847"
Sep 16 13:45:35.711236 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:45:35.711220    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847\": container with ID starting with 5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847 not found: ID does not exist" containerID="5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847"
Sep 16 13:45:35.711309 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.711240    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847"} err="failed to get container status \"5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847\": rpc error: code = NotFound desc = could not find container \"5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847\": container with ID starting with 5b34ebaa3069b596014be56199f903fe98cfde6c874b53d642d9bfe6f226d847 not found: ID does not exist"
Sep 16 13:45:35.711309 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.711275    2548 scope.go:117] "RemoveContainer" containerID="4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867"
Sep 16 13:45:35.711510 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:45:35.711492    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867\": container with ID starting with 4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867 not found: ID does not exist" containerID="4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867"
Sep 16 13:45:35.711586 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.711513    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867"} err="failed to get container status \"4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867\": rpc error: code = NotFound desc = could not find container \"4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867\": container with ID starting with 4ccf267648cc555656e62bd74fdc369d483080b6a3b7c0ecbbeb75799f4ca867 not found: ID does not exist"
Sep 16 13:45:35.733742 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.733720    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:45:35.733989 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.733976    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="prometheus"
Sep 16 13:45:35.734035 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.733992    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="prometheus"
Sep 16 13:45:35.734035 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734000    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51726060-7491-449a-9591-9cc5c8bab5b1" containerName="registry"
Sep 16 13:45:35.734035 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734005    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="51726060-7491-449a-9591-9cc5c8bab5b1" containerName="registry"
Sep 16 13:45:35.734035 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734016    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="thanos-sidecar"
Sep 16 13:45:35.734035 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734025    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="thanos-sidecar"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734038    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="config-reloader"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734044    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="config-reloader"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734053    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy-thanos"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734058    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy-thanos"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734067    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734072    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734094    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy-web"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734104    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy-web"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734112    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="init-config-reloader"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734117    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="init-config-reloader"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734163    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="config-reloader"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734171    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="51726060-7491-449a-9591-9cc5c8bab5b1" containerName="registry"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734181    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="thanos-sidecar"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734188    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy-web"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734193    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy"
Sep 16 13:45:35.734195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734198    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="kube-rbac-proxy-thanos"
Sep 16 13:45:35.734662 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.734204    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" containerName="prometheus"
Sep 16 13:45:35.737396 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.737373    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.740103 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740082    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-48kr7e4rae9ba\""
Sep 16 13:45:35.740297 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740262    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 13:45:35.740387 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740323    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 13:45:35.740568 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740542    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 13:45:35.740568 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740555    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 13:45:35.740711 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740546    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 13:45:35.740711 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740555    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 13:45:35.740711 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740542    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 13:45:35.740890 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740870    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-b78wk\""
Sep 16 13:45:35.740890 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740886    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 13:45:35.741033 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.740892    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 13:45:35.741464 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.741447    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 13:45:35.741546 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.741469    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 13:45:35.741546 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.741486    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 13:45:35.741546 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.741493    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 13:45:35.747187 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.743335    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 13:45:35.749503 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.749483    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 13:45:35.758984 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.758963    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:45:35.830416 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830391    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830518 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830420    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830518 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830437    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830518 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830451    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830518 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830472    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830660 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830522    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830660 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830544    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830660 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830568    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830660 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830585    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-config\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830660 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830607    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830660 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830629    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830843 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830665    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwlk\" (UniqueName: \"kubernetes.io/projected/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-kube-api-access-wjwlk\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830843 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830723    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830843 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830740    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830843 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830758    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830843 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830777    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830843 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830795    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.830843 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830829    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.831093 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830852    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.831093 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.830870    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931468 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931404    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931468 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931437    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931468 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931459    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwlk\" (UniqueName: \"kubernetes.io/projected/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-kube-api-access-wjwlk\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931657 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931496    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931657 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931521    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931657 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931546    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931657 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931572    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931869 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931849    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931919 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931890    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931971 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931933    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.931971 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931964    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932078 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.931991    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932078 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932020    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932078 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932048    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932078 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932072    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932293 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932096    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932293 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932130    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932293 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932154    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932293 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932180    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932293 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932191    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932293 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932206    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-config\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932591 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932310    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932591 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932352    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.932935 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.932911    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.933699 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.933676    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.934865 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.934651    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.934865 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.934725    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-config\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.934865 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.934758    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.935135 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.935032    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.935135 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.935117    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.935298 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.935142    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.935601 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.935542    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.935682 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.935607    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.935983 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.935958    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.936147 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.936126    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.937018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.936992    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.937217 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.937197    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.937217 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.937208    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.937339 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.937273    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.941351 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.941335    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwlk\" (UniqueName: \"kubernetes.io/projected/c1b1f6d4-1d74-4478-9e51-acd79d0ec89c-kube-api-access-wjwlk\") pod \"prometheus-k8s-0\" (UID: \"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:35.968812 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:35.968790    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="789c7c94-9d42-4082-9003-d160aa4a2ec9" path="/var/lib/kubelet/pods/789c7c94-9d42-4082-9003-d160aa4a2ec9/volumes"
Sep 16 13:45:36.048948 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:36.048924    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:45:36.330342 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:36.330265    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 13:45:36.333676 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:45:36.333648    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b1f6d4_1d74_4478_9e51_acd79d0ec89c.slice/crio-07fb8452379a8f8d03f9d1c05a7734d3716b960105d4398e938082ed590834ba WatchSource:0}: Error finding container 07fb8452379a8f8d03f9d1c05a7734d3716b960105d4398e938082ed590834ba: Status 404 returned error can't find the container with id 07fb8452379a8f8d03f9d1c05a7734d3716b960105d4398e938082ed590834ba
Sep 16 13:45:36.668439 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:36.668408    2548 generic.go:356] "Generic (PLEG): container finished" podID="c1b1f6d4-1d74-4478-9e51-acd79d0ec89c" containerID="10e0cc1337d7b6436fb682f82b376cab4ede53b18da4cd444f75a36f6b912b94" exitCode=0
Sep 16 13:45:36.668800 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:36.668476    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c","Type":"ContainerDied","Data":"10e0cc1337d7b6436fb682f82b376cab4ede53b18da4cd444f75a36f6b912b94"}
Sep 16 13:45:36.668800 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:36.668526    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c","Type":"ContainerStarted","Data":"07fb8452379a8f8d03f9d1c05a7734d3716b960105d4398e938082ed590834ba"}
Sep 16 13:45:37.675044 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:37.675000    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c","Type":"ContainerStarted","Data":"c484ca783ca33669cc40e0f955b377a999e59f6614f052cc96d4addba3ee2d5b"}
Sep 16 13:45:37.675471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:37.675116    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c","Type":"ContainerStarted","Data":"296a24a0918b95b93c7f2ece353852f748899a8f1454935ea7e62962336dbe72"}
Sep 16 13:45:37.675471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:37.675162    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c","Type":"ContainerStarted","Data":"5cef8cf46d818d4440a568a25397080719f886ca46471c9949b061c9c8991bdf"}
Sep 16 13:45:37.675471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:37.675199    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c","Type":"ContainerStarted","Data":"27f74a5b15e1c453746fafaae89a1deb1ae634c38b87cd82b50ca0ba49f2cace"}
Sep 16 13:45:37.675471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:37.675233    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c","Type":"ContainerStarted","Data":"03416dbfaf7f1226c8633679ee93b09441b9094a91587a379c10cde7348ced20"}
Sep 16 13:45:37.675471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:37.675289    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1b1f6d4-1d74-4478-9e51-acd79d0ec89c","Type":"ContainerStarted","Data":"f49e4ea86e425bb47ddf452184a63d228f01ab440388bc79fefcbef7e3241e67"}
Sep 16 13:45:37.705940 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:37.705888    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.684172762 podStartE2EDuration="2.705870264s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.697502ms" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:37.704424639 +0000 UTC m=+160.331906012" watchObservedRunningTime="2026-09-16 13:45:37.705870264 +0000 UTC m=+160.333351640"
Sep 16 13:45:41.049264 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:45:41.049221    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:46:36.049162 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:46:36.049130    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:46:36.063961 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:46:36.063940    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:46:36.894028 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:46:36.894001    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 13:47:36.947076 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:36.947039    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"]
Sep 16 13:47:36.949308 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:36.949292    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:36.952359 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:36.952335    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 13:47:36.953104 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:36.953089    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 13:47:36.953240 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:36.953208    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 13:47:36.953345 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:36.953232    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n7f75\""
Sep 16 13:47:36.953345 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:36.953282    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 13:47:36.965907 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:36.965885    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"]
Sep 16 13:47:37.086184 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.086149    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb874d95-6b2b-41eb-acab-aaff3c882a3e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.086562 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.086188    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb874d95-6b2b-41eb-acab-aaff3c882a3e-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.086562 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.086243    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb874d95-6b2b-41eb-acab-aaff3c882a3e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.086562 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.086302    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb874d95-6b2b-41eb-acab-aaff3c882a3e-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.086562 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.086368    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cnk8\" (UniqueName: \"kubernetes.io/projected/bb874d95-6b2b-41eb-acab-aaff3c882a3e-kube-api-access-6cnk8\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.187607 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.187573    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb874d95-6b2b-41eb-acab-aaff3c882a3e-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.187761 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.187619    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb874d95-6b2b-41eb-acab-aaff3c882a3e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.187761 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.187732    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb874d95-6b2b-41eb-acab-aaff3c882a3e-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.187877 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.187822    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cnk8\" (UniqueName: \"kubernetes.io/projected/bb874d95-6b2b-41eb-acab-aaff3c882a3e-kube-api-access-6cnk8\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.187877 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.187856    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb874d95-6b2b-41eb-acab-aaff3c882a3e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.188006 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.187986    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb874d95-6b2b-41eb-acab-aaff3c882a3e-manifests\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.189951 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.189931    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb874d95-6b2b-41eb-acab-aaff3c882a3e-tmp\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.190204 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.190184    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb874d95-6b2b-41eb-acab-aaff3c882a3e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.190285 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.190193    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb874d95-6b2b-41eb-acab-aaff3c882a3e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.197321 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.197268    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cnk8\" (UniqueName: \"kubernetes.io/projected/bb874d95-6b2b-41eb-acab-aaff3c882a3e-kube-api-access-6cnk8\") pod \"opendatahub-operator-controller-manager-f5fcfb689-h8t7z\" (UID: \"bb874d95-6b2b-41eb-acab-aaff3c882a3e\") " pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.259029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.259007    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:37.532663 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:37.532637    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"]
Sep 16 13:47:37.534226 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:47:37.534208    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb874d95_6b2b_41eb_acab_aaff3c882a3e.slice/crio-47fd17d252fa9cf91b448b1929eca4bb04571916aefac774073d2184707722e9 WatchSource:0}: Error finding container 47fd17d252fa9cf91b448b1929eca4bb04571916aefac774073d2184707722e9: Status 404 returned error can't find the container with id 47fd17d252fa9cf91b448b1929eca4bb04571916aefac774073d2184707722e9
Sep 16 13:47:38.079713 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:38.079668    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z" event={"ID":"bb874d95-6b2b-41eb-acab-aaff3c882a3e","Type":"ContainerStarted","Data":"47fd17d252fa9cf91b448b1929eca4bb04571916aefac774073d2184707722e9"}
Sep 16 13:47:41.093128 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:41.093084    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z" event={"ID":"bb874d95-6b2b-41eb-acab-aaff3c882a3e","Type":"ContainerStarted","Data":"6d4d0a637a7c69e729d0ddc486771d7203f718f99b80c7d09b3edc5e181df919"}
Sep 16 13:47:42.098678 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:42.098644    2548 generic.go:356] "Generic (PLEG): container finished" podID="bb874d95-6b2b-41eb-acab-aaff3c882a3e" containerID="6d4d0a637a7c69e729d0ddc486771d7203f718f99b80c7d09b3edc5e181df919" exitCode=0
Sep 16 13:47:42.099069 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:42.098687    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z" event={"ID":"bb874d95-6b2b-41eb-acab-aaff3c882a3e","Type":"ContainerDied","Data":"6d4d0a637a7c69e729d0ddc486771d7203f718f99b80c7d09b3edc5e181df919"}
Sep 16 13:47:43.104259 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:43.104158    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z" event={"ID":"bb874d95-6b2b-41eb-acab-aaff3c882a3e","Type":"ContainerStarted","Data":"0cf75f0d60f6931adef4dcf41ca3a5487cffef726254c5f883a5e538e4ecdc2b"}
Sep 16 13:47:43.104696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:43.104289    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:43.129655 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:43.129613    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z" podStartSLOduration=2.538588685 podStartE2EDuration="7.129600117s" totalImagesPullingTime="3.531003494s" totalInitContainerRuntime="1.060007938s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:43.128286171 +0000 UTC m=+285.755767541" watchObservedRunningTime="2026-09-16 13:47:43.129600117 +0000 UTC m=+285.757081506"
Sep 16 13:47:54.110481 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:54.110450    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f5fcfb689-h8t7z"
Sep 16 13:47:57.753500 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:57.753457    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:47:57.757535 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:57.757509    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:47:57.760590 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:47:57.760571    2548 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 13:48:07.785453 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.785416    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-257vw"]
Sep 16 13:48:07.788676 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.788653    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:07.791421 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.791399    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vztp9\""
Sep 16 13:48:07.791529 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.791474    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 13:48:07.799096 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.799073    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-257vw"]
Sep 16 13:48:07.812772 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.812746    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzdh\" (UniqueName: \"kubernetes.io/projected/8252f0a8-eaec-4b91-8ca5-2350f5a31405-kube-api-access-5fzdh\") pod \"model-serving-api-54d89bd5fc-257vw\" (UID: \"8252f0a8-eaec-4b91-8ca5-2350f5a31405\") " pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:07.812904 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.812843    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252f0a8-eaec-4b91-8ca5-2350f5a31405-tls-certs\") pod \"model-serving-api-54d89bd5fc-257vw\" (UID: \"8252f0a8-eaec-4b91-8ca5-2350f5a31405\") " pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:07.913497 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.913466    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252f0a8-eaec-4b91-8ca5-2350f5a31405-tls-certs\") pod \"model-serving-api-54d89bd5fc-257vw\" (UID: \"8252f0a8-eaec-4b91-8ca5-2350f5a31405\") " pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:07.913672 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.913536    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzdh\" (UniqueName: \"kubernetes.io/projected/8252f0a8-eaec-4b91-8ca5-2350f5a31405-kube-api-access-5fzdh\") pod \"model-serving-api-54d89bd5fc-257vw\" (UID: \"8252f0a8-eaec-4b91-8ca5-2350f5a31405\") " pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:07.913727 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:48:07.913673    2548 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 13:48:07.913772 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:48:07.913749    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8252f0a8-eaec-4b91-8ca5-2350f5a31405-tls-certs podName:8252f0a8-eaec-4b91-8ca5-2350f5a31405 nodeName:}" failed. No retries permitted until 2026-09-16 13:48:08.413734029 +0000 UTC m=+311.041215394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8252f0a8-eaec-4b91-8ca5-2350f5a31405-tls-certs") pod "model-serving-api-54d89bd5fc-257vw" (UID: "8252f0a8-eaec-4b91-8ca5-2350f5a31405") : secret "model-serving-api-tls" not found
Sep 16 13:48:07.923365 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:07.923332    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzdh\" (UniqueName: \"kubernetes.io/projected/8252f0a8-eaec-4b91-8ca5-2350f5a31405-kube-api-access-5fzdh\") pod \"model-serving-api-54d89bd5fc-257vw\" (UID: \"8252f0a8-eaec-4b91-8ca5-2350f5a31405\") " pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:08.421274 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:08.420613    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252f0a8-eaec-4b91-8ca5-2350f5a31405-tls-certs\") pod \"model-serving-api-54d89bd5fc-257vw\" (UID: \"8252f0a8-eaec-4b91-8ca5-2350f5a31405\") " pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:08.421274 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:08.420726    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8252f0a8-eaec-4b91-8ca5-2350f5a31405-tls-certs\") pod \"model-serving-api-54d89bd5fc-257vw\" (UID: \"8252f0a8-eaec-4b91-8ca5-2350f5a31405\") " pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:08.701297 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:08.701198    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:08.969516 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:08.969492    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-257vw"]
Sep 16 13:48:08.971619 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:48:08.971590    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8252f0a8_eaec_4b91_8ca5_2350f5a31405.slice/crio-05a65dec4d440eb72c6e35069b60af67cdee44e120c8dfe5057efc50881193b3 WatchSource:0}: Error finding container 05a65dec4d440eb72c6e35069b60af67cdee44e120c8dfe5057efc50881193b3: Status 404 returned error can't find the container with id 05a65dec4d440eb72c6e35069b60af67cdee44e120c8dfe5057efc50881193b3
Sep 16 13:48:08.973478 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:08.973460    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:48:09.194168 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:09.194130    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-257vw" event={"ID":"8252f0a8-eaec-4b91-8ca5-2350f5a31405","Type":"ContainerStarted","Data":"05a65dec4d440eb72c6e35069b60af67cdee44e120c8dfe5057efc50881193b3"}
Sep 16 13:48:12.206978 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:12.206936    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-257vw" event={"ID":"8252f0a8-eaec-4b91-8ca5-2350f5a31405","Type":"ContainerStarted","Data":"4866c8dc3e62c8046ddf6f72d8b9670450fa10783126c467a24bf2b73e76a69e"}
Sep 16 13:48:12.207381 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:12.207046    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:12.225769 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:12.225722    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-257vw" podStartSLOduration=2.967374804 podStartE2EDuration="5.225706605s" totalImagesPullingTime="2.258331801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:12.224153412 +0000 UTC m=+314.851634798" watchObservedRunningTime="2026-09-16 13:48:12.225706605 +0000 UTC m=+314.853187978"
Sep 16 13:48:16.365603 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.365563    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"]
Sep 16 13:48:16.368242 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.368219    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.430743 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.430714    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 13:48:16.431374 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.431345    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 13:48:16.432498 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.432478    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 13:48:16.432498 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.432490    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 13:48:16.432687 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.432490    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7kb6t\""
Sep 16 13:48:16.432687 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.432531    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:48:16.435683 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.435659    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"]
Sep 16 13:48:16.488730 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.488708    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b2df2b7-b873-42cf-a6a0-84be2e90c057-manager-config\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.488892 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.488735    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljxc\" (UniqueName: \"kubernetes.io/projected/5b2df2b7-b873-42cf-a6a0-84be2e90c057-kube-api-access-8ljxc\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.488892 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.488757    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b2df2b7-b873-42cf-a6a0-84be2e90c057-cert\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.488892 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.488780    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2df2b7-b873-42cf-a6a0-84be2e90c057-metrics-cert\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.589611 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.589581    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b2df2b7-b873-42cf-a6a0-84be2e90c057-manager-config\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.589731 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.589614    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljxc\" (UniqueName: \"kubernetes.io/projected/5b2df2b7-b873-42cf-a6a0-84be2e90c057-kube-api-access-8ljxc\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.589731 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.589642    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b2df2b7-b873-42cf-a6a0-84be2e90c057-cert\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.589809 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.589748    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2df2b7-b873-42cf-a6a0-84be2e90c057-metrics-cert\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.590428 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.590404    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5b2df2b7-b873-42cf-a6a0-84be2e90c057-manager-config\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.592205 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.592182    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b2df2b7-b873-42cf-a6a0-84be2e90c057-cert\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.592316 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.592223    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2df2b7-b873-42cf-a6a0-84be2e90c057-metrics-cert\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.612298 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.612274    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljxc\" (UniqueName: \"kubernetes.io/projected/5b2df2b7-b873-42cf-a6a0-84be2e90c057-kube-api-access-8ljxc\") pod \"lws-controller-manager-6465fbddb8-dxpw8\" (UID: \"5b2df2b7-b873-42cf-a6a0-84be2e90c057\") " pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.654316 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.654221    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"]
Sep 16 13:48:16.656323 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.656299    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.662282 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.662258    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 13:48:16.662426 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.662289    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b64jc\""
Sep 16 13:48:16.662548 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.662530    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:48:16.662642 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.662616    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 13:48:16.677384 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.677363    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:16.688192 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.688170    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"]
Sep 16 13:48:16.791153 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791115    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.791350 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791167    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.791350 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791228    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.791350 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791295    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.791350 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791326    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.791523 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791381    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.791523 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791408    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae21a4ff-f37f-43cd-876d-1351113bb384-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.791523 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791441    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.791523 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.791459    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz48z\" (UniqueName: \"kubernetes.io/projected/ae21a4ff-f37f-43cd-876d-1351113bb384-kube-api-access-pz48z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.891946 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.891908    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892119 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.891965    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892119 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892098    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892235 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892154    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892235 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892203    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae21a4ff-f37f-43cd-876d-1351113bb384-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892368 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892240    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892368 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892284    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz48z\" (UniqueName: \"kubernetes.io/projected/ae21a4ff-f37f-43cd-876d-1351113bb384-kube-api-access-pz48z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892375    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892411    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892445    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892620 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892496    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892620 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892579    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.892887 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.892748    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.893191 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.893140    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae21a4ff-f37f-43cd-876d-1351113bb384-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.894750 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.894726    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.895059 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.895036    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.906994 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.906939    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae21a4ff-f37f-43cd-876d-1351113bb384-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.907157 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.907138    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz48z\" (UniqueName: \"kubernetes.io/projected/ae21a4ff-f37f-43cd-876d-1351113bb384-kube-api-access-pz48z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq\" (UID: \"ae21a4ff-f37f-43cd-876d-1351113bb384\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:16.960371 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.960347    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"]
Sep 16 13:48:16.962362 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:48:16.962335    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2df2b7_b873_42cf_a6a0_84be2e90c057.slice/crio-99f202b61db03fa4d932d0a769fc860c5d47d0f09c567f098c0e264825f0d57c WatchSource:0}: Error finding container 99f202b61db03fa4d932d0a769fc860c5d47d0f09c567f098c0e264825f0d57c: Status 404 returned error can't find the container with id 99f202b61db03fa4d932d0a769fc860c5d47d0f09c567f098c0e264825f0d57c
Sep 16 13:48:16.968238 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:16.968217    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:17.226787 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:17.226713    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8" event={"ID":"5b2df2b7-b873-42cf-a6a0-84be2e90c057","Type":"ContainerStarted","Data":"99f202b61db03fa4d932d0a769fc860c5d47d0f09c567f098c0e264825f0d57c"}
Sep 16 13:48:17.270180 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:17.270037    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"]
Sep 16 13:48:17.272496 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:48:17.272463    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae21a4ff_f37f_43cd_876d_1351113bb384.slice/crio-0842d10be471864eb20b8caf4ffcd1486ab52497a2beb6a798aadb33829a6bf4 WatchSource:0}: Error finding container 0842d10be471864eb20b8caf4ffcd1486ab52497a2beb6a798aadb33829a6bf4: Status 404 returned error can't find the container with id 0842d10be471864eb20b8caf4ffcd1486ab52497a2beb6a798aadb33829a6bf4
Sep 16 13:48:18.232529 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:18.232484    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq" event={"ID":"ae21a4ff-f37f-43cd-876d-1351113bb384","Type":"ContainerStarted","Data":"0842d10be471864eb20b8caf4ffcd1486ab52497a2beb6a798aadb33829a6bf4"}
Sep 16 13:48:19.311477 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.311440    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"]
Sep 16 13:48:19.314238 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.314216    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:19.317005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.316832    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 13:48:19.317005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.316847    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ksh27\""
Sep 16 13:48:19.333536 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.333502    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"]
Sep 16 13:48:19.414457 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.414421    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94bc5768-cee6-4110-b0d1-9b5826bdb8f0-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwcrn\" (UID: \"94bc5768-cee6-4110-b0d1-9b5826bdb8f0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:19.414638 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.414605    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nq24\" (UniqueName: \"kubernetes.io/projected/94bc5768-cee6-4110-b0d1-9b5826bdb8f0-kube-api-access-4nq24\") pod \"kserve-controller-manager-d6f4b65b8-xwcrn\" (UID: \"94bc5768-cee6-4110-b0d1-9b5826bdb8f0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:19.516201 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.515974    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nq24\" (UniqueName: \"kubernetes.io/projected/94bc5768-cee6-4110-b0d1-9b5826bdb8f0-kube-api-access-4nq24\") pod \"kserve-controller-manager-d6f4b65b8-xwcrn\" (UID: \"94bc5768-cee6-4110-b0d1-9b5826bdb8f0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:19.516201 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.516030    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94bc5768-cee6-4110-b0d1-9b5826bdb8f0-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwcrn\" (UID: \"94bc5768-cee6-4110-b0d1-9b5826bdb8f0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:19.519934 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.519382    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94bc5768-cee6-4110-b0d1-9b5826bdb8f0-cert\") pod \"kserve-controller-manager-d6f4b65b8-xwcrn\" (UID: \"94bc5768-cee6-4110-b0d1-9b5826bdb8f0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:19.532059 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.532031    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nq24\" (UniqueName: \"kubernetes.io/projected/94bc5768-cee6-4110-b0d1-9b5826bdb8f0-kube-api-access-4nq24\") pod \"kserve-controller-manager-d6f4b65b8-xwcrn\" (UID: \"94bc5768-cee6-4110-b0d1-9b5826bdb8f0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:19.629126 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:19.629091    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:20.866194 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:20.866148    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"]
Sep 16 13:48:20.869198 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:48:20.869172    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bc5768_cee6_4110_b0d1_9b5826bdb8f0.slice/crio-fc1ca4dcbdff04999e3f46b0c9ba98a26edfc75bb97903c586306365a96ff347 WatchSource:0}: Error finding container fc1ca4dcbdff04999e3f46b0c9ba98a26edfc75bb97903c586306365a96ff347: Status 404 returned error can't find the container with id fc1ca4dcbdff04999e3f46b0c9ba98a26edfc75bb97903c586306365a96ff347
Sep 16 13:48:21.248030 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:21.247948    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8" event={"ID":"5b2df2b7-b873-42cf-a6a0-84be2e90c057","Type":"ContainerStarted","Data":"85793d0ac3045c3092ee9640d24387b2d6e10d9dc875d97b7a028eb8a35d04d8"}
Sep 16 13:48:21.248030 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:21.248013    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:21.249613 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:21.249586    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn" event={"ID":"94bc5768-cee6-4110-b0d1-9b5826bdb8f0","Type":"ContainerStarted","Data":"fc1ca4dcbdff04999e3f46b0c9ba98a26edfc75bb97903c586306365a96ff347"}
Sep 16 13:48:21.272499 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:21.272453    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8" podStartSLOduration=1.5729378569999999 podStartE2EDuration="5.272433433s" totalImagesPullingTime="3.699495576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:21.27085068 +0000 UTC m=+323.898332254" watchObservedRunningTime="2026-09-16 13:48:21.272433433 +0000 UTC m=+323.899914809"
Sep 16 13:48:23.214987 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:23.214956    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-257vw"
Sep 16 13:48:23.481975 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:23.481931    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:48:23.482065 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:23.482016    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:48:23.482065 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:23.482046    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:48:24.269846 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:24.269803    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq" event={"ID":"ae21a4ff-f37f-43cd-876d-1351113bb384","Type":"ContainerStarted","Data":"7cff9f92d8231804c3834a2b0e4e7a436d16bd0f64db428d8b1af226fe99b8c4"}
Sep 16 13:48:24.271233 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:24.271205    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn" event={"ID":"94bc5768-cee6-4110-b0d1-9b5826bdb8f0","Type":"ContainerStarted","Data":"d81f407e9a25ce1da2a2450e443e37a6c6f7780c8d66b3d7138b97a9133fad9f"}
Sep 16 13:48:24.271355 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:24.271281    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:48:24.291372 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:24.291331    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq" podStartSLOduration=2.083991025 podStartE2EDuration="8.2913193s" totalImagesPullingTime="6.207328275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:24.290422836 +0000 UTC m=+326.917904203" watchObservedRunningTime="2026-09-16 13:48:24.2913193 +0000 UTC m=+326.918800673"
Sep 16 13:48:24.307889 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:24.307851    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn" podStartSLOduration=2.696357149 podStartE2EDuration="5.30783782s" totalImagesPullingTime="2.611480671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:24.306324436 +0000 UTC m=+326.933805840" watchObservedRunningTime="2026-09-16 13:48:24.30783782 +0000 UTC m=+326.935319194"
Sep 16 13:48:24.968964 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:24.968922    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:24.973543 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:24.973519    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:25.275150 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:25.275065    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:25.275803 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:25.275784    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq"
Sep 16 13:48:32.256473 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:32.256438    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6465fbddb8-dxpw8"
Sep 16 13:48:55.279832 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:48:55.279761    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xwcrn"
Sep 16 13:49:33.006705 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.006671    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"]
Sep 16 13:49:33.009984 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.009968    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"
Sep 16 13:49:33.013386 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.013363    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-m55t6\""
Sep 16 13:49:33.013904 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.013879    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 13:49:33.014007 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.013906    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 13:49:33.014302 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.014288    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 13:49:33.021629 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.021607    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"]
Sep 16 13:49:33.083670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.083638    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgk7j\" (UniqueName: \"kubernetes.io/projected/4081aa12-adb5-41e0-bded-1eea778917f9-kube-api-access-dgk7j\") pod \"dns-operator-controller-manager-7b84cd7c9c-swdf7\" (UID: \"4081aa12-adb5-41e0-bded-1eea778917f9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"
Sep 16 13:49:33.184833 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.184784    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgk7j\" (UniqueName: \"kubernetes.io/projected/4081aa12-adb5-41e0-bded-1eea778917f9-kube-api-access-dgk7j\") pod \"dns-operator-controller-manager-7b84cd7c9c-swdf7\" (UID: \"4081aa12-adb5-41e0-bded-1eea778917f9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"
Sep 16 13:49:33.199326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.199294    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgk7j\" (UniqueName: \"kubernetes.io/projected/4081aa12-adb5-41e0-bded-1eea778917f9-kube-api-access-dgk7j\") pod \"dns-operator-controller-manager-7b84cd7c9c-swdf7\" (UID: \"4081aa12-adb5-41e0-bded-1eea778917f9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"
Sep 16 13:49:33.320345 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.320242    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"
Sep 16 13:49:33.590260 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:33.590221    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"]
Sep 16 13:49:33.592807 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:49:33.592782    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4081aa12_adb5_41e0_bded_1eea778917f9.slice/crio-b29de55b48536a6e5455dd1af14da763d353c74f9dcff08fb4477b2ac0367e76 WatchSource:0}: Error finding container b29de55b48536a6e5455dd1af14da763d353c74f9dcff08fb4477b2ac0367e76: Status 404 returned error can't find the container with id b29de55b48536a6e5455dd1af14da763d353c74f9dcff08fb4477b2ac0367e76
Sep 16 13:49:34.528225 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:34.528187    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7" event={"ID":"4081aa12-adb5-41e0-bded-1eea778917f9","Type":"ContainerStarted","Data":"b29de55b48536a6e5455dd1af14da763d353c74f9dcff08fb4477b2ac0367e76"}
Sep 16 13:49:36.245083 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.244992    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p7js2"]
Sep 16 13:49:36.249659 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.249640    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p7js2"
Sep 16 13:49:36.266512 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.266491    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5j6ks\""
Sep 16 13:49:36.272846 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.272823    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p7js2"]
Sep 16 13:49:36.313017 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.312986    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cftq2\" (UniqueName: \"kubernetes.io/projected/0561a9a3-4b3f-4d21-8aa6-aa76148f90d1-kube-api-access-cftq2\") pod \"authorino-operator-76b478bd78-p7js2\" (UID: \"0561a9a3-4b3f-4d21-8aa6-aa76148f90d1\") " pod="kuadrant-system/authorino-operator-76b478bd78-p7js2"
Sep 16 13:49:36.414044 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.414014    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cftq2\" (UniqueName: \"kubernetes.io/projected/0561a9a3-4b3f-4d21-8aa6-aa76148f90d1-kube-api-access-cftq2\") pod \"authorino-operator-76b478bd78-p7js2\" (UID: \"0561a9a3-4b3f-4d21-8aa6-aa76148f90d1\") " pod="kuadrant-system/authorino-operator-76b478bd78-p7js2"
Sep 16 13:49:36.426758 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.426738    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cftq2\" (UniqueName: \"kubernetes.io/projected/0561a9a3-4b3f-4d21-8aa6-aa76148f90d1-kube-api-access-cftq2\") pod \"authorino-operator-76b478bd78-p7js2\" (UID: \"0561a9a3-4b3f-4d21-8aa6-aa76148f90d1\") " pod="kuadrant-system/authorino-operator-76b478bd78-p7js2"
Sep 16 13:49:36.560800 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.560710    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p7js2"
Sep 16 13:49:36.867072 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:36.867048    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p7js2"]
Sep 16 13:49:36.868903 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:49:36.868875    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0561a9a3_4b3f_4d21_8aa6_aa76148f90d1.slice/crio-556864199b683e5cb6438e64cfae3ea57174e916e5a4a38c1db8e9505cbcd9e6 WatchSource:0}: Error finding container 556864199b683e5cb6438e64cfae3ea57174e916e5a4a38c1db8e9505cbcd9e6: Status 404 returned error can't find the container with id 556864199b683e5cb6438e64cfae3ea57174e916e5a4a38c1db8e9505cbcd9e6
Sep 16 13:49:37.542939 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:37.542894    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p7js2" event={"ID":"0561a9a3-4b3f-4d21-8aa6-aa76148f90d1","Type":"ContainerStarted","Data":"556864199b683e5cb6438e64cfae3ea57174e916e5a4a38c1db8e9505cbcd9e6"}
Sep 16 13:49:39.552888 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:39.552798    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p7js2" event={"ID":"0561a9a3-4b3f-4d21-8aa6-aa76148f90d1","Type":"ContainerStarted","Data":"4451e5ad94ecf2f9cb4e51d49e3242dc28d9ac17586fcbbf6eaed54ecba99bc1"}
Sep 16 13:49:39.553235 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:39.552927    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-p7js2"
Sep 16 13:49:39.581770 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:39.581721    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-p7js2" podStartSLOduration=1.282420925 podStartE2EDuration="3.581705952s" totalImagesPullingTime="2.299285027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:39.579150945 +0000 UTC m=+402.206632319" watchObservedRunningTime="2026-09-16 13:49:39.581705952 +0000 UTC m=+402.209187325"
Sep 16 13:49:40.894680 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:40.894642    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"]
Sep 16 13:49:40.898318 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:40.898296    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:40.902014 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:40.901987    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-khthc\""
Sep 16 13:49:40.913679 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:40.913649    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"]
Sep 16 13:49:40.964214 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:40.964146    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjnd\" (UniqueName: \"kubernetes.io/projected/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-kube-api-access-qrjnd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" (UID: \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:40.964214 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:40.964211    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" (UID: \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:41.065118 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.065081    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjnd\" (UniqueName: \"kubernetes.io/projected/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-kube-api-access-qrjnd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" (UID: \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:41.065326 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.065141    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" (UID: \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:41.065552 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.065527    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" (UID: \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:41.075504 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.075486    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjnd\" (UniqueName: \"kubernetes.io/projected/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-kube-api-access-qrjnd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" (UID: \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:41.211876 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.211792    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:41.562191 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.562156    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7" event={"ID":"4081aa12-adb5-41e0-bded-1eea778917f9","Type":"ContainerStarted","Data":"a9d17bce793aa0779f96450490c7298db36f60d54b4ebd880fa0504415ffc7c5"}
Sep 16 13:49:41.562342 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.562305    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"
Sep 16 13:49:41.583305 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.583233    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7" podStartSLOduration=1.748071918 podStartE2EDuration="9.583211082s" totalImagesPullingTime="7.835139164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:41.582505527 +0000 UTC m=+404.209986904" watchObservedRunningTime="2026-09-16 13:49:41.583211082 +0000 UTC m=+404.210692457"
Sep 16 13:49:41.637316 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:41.637285    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"]
Sep 16 13:49:41.640743 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:49:41.640715    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7663ac1_ba0e_42d9_80e1_fa5d05bbc491.slice/crio-4ceb39f8670f36f0530225bc473fc9ac07e7c4b991b167af41f9c412fd1abe70 WatchSource:0}: Error finding container 4ceb39f8670f36f0530225bc473fc9ac07e7c4b991b167af41f9c412fd1abe70: Status 404 returned error can't find the container with id 4ceb39f8670f36f0530225bc473fc9ac07e7c4b991b167af41f9c412fd1abe70
Sep 16 13:49:42.569368 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:42.569323    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr" event={"ID":"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491","Type":"ContainerStarted","Data":"4ceb39f8670f36f0530225bc473fc9ac07e7c4b991b167af41f9c412fd1abe70"}
Sep 16 13:49:45.586114 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:45.586069    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr" event={"ID":"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491","Type":"ContainerStarted","Data":"9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184"}
Sep 16 13:49:45.586489 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:45.586275    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:45.613357 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:45.613308    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr" podStartSLOduration=2.222121406 podStartE2EDuration="5.61328886s" totalImagesPullingTime="3.391167454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:45.611446471 +0000 UTC m=+408.238927845" watchObservedRunningTime="2026-09-16 13:49:45.61328886 +0000 UTC m=+408.240770255"
Sep 16 13:49:50.558964 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:50.558932    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-p7js2"
Sep 16 13:49:52.571753 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:52.571712    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-swdf7"
Sep 16 13:49:56.592821 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:56.592787    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:58.879021 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.878984    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"]
Sep 16 13:49:58.879484 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.879175    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" containerName="manager" containerID="cri-o://9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184" gracePeriod=2
Sep 16 13:49:58.890854 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.890826    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"]
Sep 16 13:49:58.906288 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:49:58.906257    2548 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\": pods \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:58.919533 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.919506    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"]
Sep 16 13:49:58.919914 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.919897    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" containerName="manager"
Sep 16 13:49:58.919995 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.919915    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" containerName="manager"
Sep 16 13:49:58.919995 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.919983    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" containerName="manager"
Sep 16 13:49:58.923636 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.923618    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:49:58.928678 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:49:58.928644    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:58.944049 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:58.944025    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"]
Sep 16 13:49:59.016360 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.016329    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgf9j\" (UniqueName: \"kubernetes.io/projected/52415408-fcd6-4c6f-ab3f-be8f19b344f9-kube-api-access-fgf9j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwpng\" (UID: \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:49:59.016517 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.016486    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52415408-fcd6-4c6f-ab3f-be8f19b344f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwpng\" (UID: \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:49:59.116872 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.116847    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52415408-fcd6-4c6f-ab3f-be8f19b344f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwpng\" (UID: \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:49:59.117090 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.117075    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgf9j\" (UniqueName: \"kubernetes.io/projected/52415408-fcd6-4c6f-ab3f-be8f19b344f9-kube-api-access-fgf9j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwpng\" (UID: \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:49:59.117263 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.117221    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52415408-fcd6-4c6f-ab3f-be8f19b344f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwpng\" (UID: \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:49:59.119334 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.119320    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:59.121470 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:49:59.121446    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:59.126419 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.126400    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgf9j\" (UniqueName: \"kubernetes.io/projected/52415408-fcd6-4c6f-ab3f-be8f19b344f9-kube-api-access-fgf9j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwpng\" (UID: \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:49:59.217475 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.217417    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrjnd\" (UniqueName: \"kubernetes.io/projected/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-kube-api-access-qrjnd\") pod \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\" (UID: \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\") "
Sep 16 13:49:59.217598 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.217510    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-extensions-socket-volume\") pod \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\" (UID: \"b7663ac1-ba0e-42d9-80e1-fa5d05bbc491\") "
Sep 16 13:49:59.217835 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.217814    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" (UID: "b7663ac1-ba0e-42d9-80e1-fa5d05bbc491"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:59.219469 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.219443    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-kube-api-access-qrjnd" (OuterVolumeSpecName: "kube-api-access-qrjnd") pod "b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" (UID: "b7663ac1-ba0e-42d9-80e1-fa5d05bbc491"). InnerVolumeSpecName "kube-api-access-qrjnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:49:59.256311 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.256288    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:49:59.318040 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.318014    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrjnd\" (UniqueName: \"kubernetes.io/projected/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-kube-api-access-qrjnd\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:49:59.318040 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.318039    2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491-extensions-socket-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:49:59.550718 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.550644    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"]
Sep 16 13:49:59.554441 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:49:59.554409    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52415408_fcd6_4c6f_ab3f_be8f19b344f9.slice/crio-ca21b4886e0c7ec1de96af9ea3e0903fbc46c9fedd14022a6b694e3d54a0ad3c WatchSource:0}: Error finding container ca21b4886e0c7ec1de96af9ea3e0903fbc46c9fedd14022a6b694e3d54a0ad3c: Status 404 returned error can't find the container with id ca21b4886e0c7ec1de96af9ea3e0903fbc46c9fedd14022a6b694e3d54a0ad3c
Sep 16 13:49:59.650192 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.650164    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng" event={"ID":"52415408-fcd6-4c6f-ab3f-be8f19b344f9","Type":"ContainerStarted","Data":"ca21b4886e0c7ec1de96af9ea3e0903fbc46c9fedd14022a6b694e3d54a0ad3c"}
Sep 16 13:49:59.651580 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.651555    2548 generic.go:356] "Generic (PLEG): container finished" podID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" containerID="9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184" exitCode=0
Sep 16 13:49:59.651679 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.651603    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:59.651679 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.651654    2548 scope.go:117] "RemoveContainer" containerID="9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184"
Sep 16 13:49:59.654792 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:49:59.654759    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:59.664673 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.664653    2548 scope.go:117] "RemoveContainer" containerID="9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184"
Sep 16 13:49:59.664980 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:49:59.664962    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184\": container with ID starting with 9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184 not found: ID does not exist" containerID="9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184"
Sep 16 13:49:59.665056 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.664993    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184"} err="failed to get container status \"9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184\": rpc error: code = NotFound desc = could not find container \"9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184\": container with ID starting with 9d7a953c4e9e1ea1c547a75e345c12424e20b9efe8d81fb100e5533ff10e5184 not found: ID does not exist"
Sep 16 13:49:59.668723 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:49:59.668699    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hnlkr\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hnlkr"
Sep 16 13:49:59.969022 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:49:59.968989    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7663ac1-ba0e-42d9-80e1-fa5d05bbc491" path="/var/lib/kubelet/pods/b7663ac1-ba0e-42d9-80e1-fa5d05bbc491/volumes"
Sep 16 13:50:00.657930 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:00.657895    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng" event={"ID":"52415408-fcd6-4c6f-ab3f-be8f19b344f9","Type":"ContainerStarted","Data":"ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b"}
Sep 16 13:50:00.658128 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:00.657969    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:50:00.681341 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:00.681297    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng" podStartSLOduration=2.6812835550000003 podStartE2EDuration="2.681283555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:00.680760954 +0000 UTC m=+423.308242333" watchObservedRunningTime="2026-09-16 13:50:00.681283555 +0000 UTC m=+423.308764929"
Sep 16 13:50:11.664814 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:11.664783    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:50:13.809018 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:13.808982    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"]
Sep 16 13:50:13.809460 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:13.809207    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng" podUID="52415408-fcd6-4c6f-ab3f-be8f19b344f9" containerName="manager" containerID="cri-o://ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b" gracePeriod=10
Sep 16 13:50:14.048951 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.048925    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:50:14.094666 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.094596    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"]
Sep 16 13:50:14.094892 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.094878    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52415408-fcd6-4c6f-ab3f-be8f19b344f9" containerName="manager"
Sep 16 13:50:14.094892 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.094891    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="52415408-fcd6-4c6f-ab3f-be8f19b344f9" containerName="manager"
Sep 16 13:50:14.095019 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.094951    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="52415408-fcd6-4c6f-ab3f-be8f19b344f9" containerName="manager"
Sep 16 13:50:14.096888 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.096868    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:14.118765 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.118738    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"]
Sep 16 13:50:14.129145 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.129126    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgf9j\" (UniqueName: \"kubernetes.io/projected/52415408-fcd6-4c6f-ab3f-be8f19b344f9-kube-api-access-fgf9j\") pod \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\" (UID: \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\") "
Sep 16 13:50:14.129242 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.129223    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52415408-fcd6-4c6f-ab3f-be8f19b344f9-extensions-socket-volume\") pod \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\" (UID: \"52415408-fcd6-4c6f-ab3f-be8f19b344f9\") "
Sep 16 13:50:14.129410 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.129389    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c74a5cfb-5249-4590-a6db-478c3dff65b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xbn67\" (UID: \"c74a5cfb-5249-4590-a6db-478c3dff65b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:14.129476 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.129443    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqnl\" (UniqueName: \"kubernetes.io/projected/c74a5cfb-5249-4590-a6db-478c3dff65b3-kube-api-access-7bqnl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xbn67\" (UID: \"c74a5cfb-5249-4590-a6db-478c3dff65b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:14.129622 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.129605    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52415408-fcd6-4c6f-ab3f-be8f19b344f9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "52415408-fcd6-4c6f-ab3f-be8f19b344f9" (UID: "52415408-fcd6-4c6f-ab3f-be8f19b344f9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:50:14.131212 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.131183    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52415408-fcd6-4c6f-ab3f-be8f19b344f9-kube-api-access-fgf9j" (OuterVolumeSpecName: "kube-api-access-fgf9j") pod "52415408-fcd6-4c6f-ab3f-be8f19b344f9" (UID: "52415408-fcd6-4c6f-ab3f-be8f19b344f9"). InnerVolumeSpecName "kube-api-access-fgf9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:50:14.230882 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.230857    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c74a5cfb-5249-4590-a6db-478c3dff65b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xbn67\" (UID: \"c74a5cfb-5249-4590-a6db-478c3dff65b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:14.230982 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.230897    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqnl\" (UniqueName: \"kubernetes.io/projected/c74a5cfb-5249-4590-a6db-478c3dff65b3-kube-api-access-7bqnl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xbn67\" (UID: \"c74a5cfb-5249-4590-a6db-478c3dff65b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:14.230982 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.230952    2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52415408-fcd6-4c6f-ab3f-be8f19b344f9-extensions-socket-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:50:14.230982 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.230963    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgf9j\" (UniqueName: \"kubernetes.io/projected/52415408-fcd6-4c6f-ab3f-be8f19b344f9-kube-api-access-fgf9j\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:50:14.231227 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.231207    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c74a5cfb-5249-4590-a6db-478c3dff65b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xbn67\" (UID: \"c74a5cfb-5249-4590-a6db-478c3dff65b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:14.244712 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.244689    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqnl\" (UniqueName: \"kubernetes.io/projected/c74a5cfb-5249-4590-a6db-478c3dff65b3-kube-api-access-7bqnl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xbn67\" (UID: \"c74a5cfb-5249-4590-a6db-478c3dff65b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:14.406801 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.406778    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:14.680287 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.680240    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"]
Sep 16 13:50:14.681635 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:50:14.681611    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74a5cfb_5249_4590_a6db_478c3dff65b3.slice/crio-cd19247ab03ab3330371f145cc057a9160bbbb6941ff817001ae1e7d14824105 WatchSource:0}: Error finding container cd19247ab03ab3330371f145cc057a9160bbbb6941ff817001ae1e7d14824105: Status 404 returned error can't find the container with id cd19247ab03ab3330371f145cc057a9160bbbb6941ff817001ae1e7d14824105
Sep 16 13:50:14.722523 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.722499    2548 generic.go:356] "Generic (PLEG): container finished" podID="52415408-fcd6-4c6f-ab3f-be8f19b344f9" containerID="ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b" exitCode=0
Sep 16 13:50:14.722624 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.722536    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"
Sep 16 13:50:14.722624 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.722582    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng" event={"ID":"52415408-fcd6-4c6f-ab3f-be8f19b344f9","Type":"ContainerDied","Data":"ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b"}
Sep 16 13:50:14.722732 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.722642    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng" event={"ID":"52415408-fcd6-4c6f-ab3f-be8f19b344f9","Type":"ContainerDied","Data":"ca21b4886e0c7ec1de96af9ea3e0903fbc46c9fedd14022a6b694e3d54a0ad3c"}
Sep 16 13:50:14.722732 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.722676    2548 scope.go:117] "RemoveContainer" containerID="ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b"
Sep 16 13:50:14.724107 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.724086    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67" event={"ID":"c74a5cfb-5249-4590-a6db-478c3dff65b3","Type":"ContainerStarted","Data":"cd19247ab03ab3330371f145cc057a9160bbbb6941ff817001ae1e7d14824105"}
Sep 16 13:50:14.732973 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.732953    2548 scope.go:117] "RemoveContainer" containerID="ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b"
Sep 16 13:50:14.733236 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:50:14.733220    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b\": container with ID starting with ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b not found: ID does not exist" containerID="ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b"
Sep 16 13:50:14.733338 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.733242    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b"} err="failed to get container status \"ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b\": rpc error: code = NotFound desc = could not find container \"ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b\": container with ID starting with ee646114dcc5144d02da3e4cc177dd536875312afa49744d8f02a780d1effb3b not found: ID does not exist"
Sep 16 13:50:14.746129 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.746106    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"]
Sep 16 13:50:14.751568 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:14.751547    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwpng"]
Sep 16 13:50:15.730763 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:15.730728    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67" event={"ID":"c74a5cfb-5249-4590-a6db-478c3dff65b3","Type":"ContainerStarted","Data":"87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f"}
Sep 16 13:50:15.731148 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:15.730841    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:15.752365 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:15.752324    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67" podStartSLOduration=1.752309932 podStartE2EDuration="1.752309932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:15.750638049 +0000 UTC m=+438.378119424" watchObservedRunningTime="2026-09-16 13:50:15.752309932 +0000 UTC m=+438.379791316"
Sep 16 13:50:15.968523 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:15.968495    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52415408-fcd6-4c6f-ab3f-be8f19b344f9" path="/var/lib/kubelet/pods/52415408-fcd6-4c6f-ab3f-be8f19b344f9/volumes"
Sep 16 13:50:26.738807 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:26.738727    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 13:50:30.458188 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.458149    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"]
Sep 16 13:50:30.461168 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.461149    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.463571 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.463547    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4mh7p\""
Sep 16 13:50:30.472587 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.472567    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"]
Sep 16 13:50:30.556206 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556166    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.556206 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556205    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.556441 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556232    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.556441 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556342    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.556441 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556378    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.556441 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556398    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8d8\" (UniqueName: \"kubernetes.io/projected/1ce50390-e6e1-49fe-9e01-8e74baea14ac-kube-api-access-6c8d8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.556441 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556418    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.556615 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556443    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.556615 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.556465    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657154 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657118    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657154 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657158    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657175    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8d8\" (UniqueName: \"kubernetes.io/projected/1ce50390-e6e1-49fe-9e01-8e74baea14ac-kube-api-access-6c8d8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657200    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657228    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657275    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657331    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657351    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657381    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657830 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657557    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657830 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657697    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657830 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657750    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.657830 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.657789    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.658126 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.658108    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.659398 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.659378    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.659686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.659670    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.666769 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.666748    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce50390-e6e1-49fe-9e01-8e74baea14ac-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.666861 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.666741    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8d8\" (UniqueName: \"kubernetes.io/projected/1ce50390-e6e1-49fe-9e01-8e74baea14ac-kube-api-access-6c8d8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-l5hkn\" (UID: \"1ce50390-e6e1-49fe-9e01-8e74baea14ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:30.772365 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:30.772288    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:31.042287 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:31.042263    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"]
Sep 16 13:50:31.044215 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:50:31.044187    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce50390_e6e1_49fe_9e01_8e74baea14ac.slice/crio-cf9f13311bc1ff34d71135e86be199cca9d338fc00dcb2fbb48f0202de4aeb48 WatchSource:0}: Error finding container cf9f13311bc1ff34d71135e86be199cca9d338fc00dcb2fbb48f0202de4aeb48: Status 404 returned error can't find the container with id cf9f13311bc1ff34d71135e86be199cca9d338fc00dcb2fbb48f0202de4aeb48
Sep 16 13:50:31.046297 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:31.046264    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:50:31.046365 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:31.046331    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:50:31.046365 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:31.046357    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:50:31.802873 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:31.802840    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" event={"ID":"1ce50390-e6e1-49fe-9e01-8e74baea14ac","Type":"ContainerStarted","Data":"1600ad8b43cb196ae96804501d250b2d672e8641a5c58d127ddf79f4ea6d8303"}
Sep 16 13:50:31.803267 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:31.802889    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" event={"ID":"1ce50390-e6e1-49fe-9e01-8e74baea14ac","Type":"ContainerStarted","Data":"cf9f13311bc1ff34d71135e86be199cca9d338fc00dcb2fbb48f0202de4aeb48"}
Sep 16 13:50:31.824835 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:31.824787    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" podStartSLOduration=1.8247717030000001 podStartE2EDuration="1.824771703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:31.82359144 +0000 UTC m=+454.451072816" watchObservedRunningTime="2026-09-16 13:50:31.824771703 +0000 UTC m=+454.452253077"
Sep 16 13:50:32.772614 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:32.772577    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:32.777208 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:32.777181    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:32.807466 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:32.807434    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:32.808298 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:32.808280    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:50:45.056527 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.056492    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-jr25v"]
Sep 16 13:50:45.060130 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.060112    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jr25v"
Sep 16 13:50:45.062574 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.062554    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jqwlz\""
Sep 16 13:50:45.065993 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.065973    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jr25v"]
Sep 16 13:50:45.176717 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.176683    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkgdv\" (UniqueName: \"kubernetes.io/projected/da8cfc6c-b123-4ee0-8775-7bf81abe9658-kube-api-access-pkgdv\") pod \"authorino-59c974bd48-jr25v\" (UID: \"da8cfc6c-b123-4ee0-8775-7bf81abe9658\") " pod="kuadrant-system/authorino-59c974bd48-jr25v"
Sep 16 13:50:45.277123 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.277088    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkgdv\" (UniqueName: \"kubernetes.io/projected/da8cfc6c-b123-4ee0-8775-7bf81abe9658-kube-api-access-pkgdv\") pod \"authorino-59c974bd48-jr25v\" (UID: \"da8cfc6c-b123-4ee0-8775-7bf81abe9658\") " pod="kuadrant-system/authorino-59c974bd48-jr25v"
Sep 16 13:50:45.287572 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.287544    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkgdv\" (UniqueName: \"kubernetes.io/projected/da8cfc6c-b123-4ee0-8775-7bf81abe9658-kube-api-access-pkgdv\") pod \"authorino-59c974bd48-jr25v\" (UID: \"da8cfc6c-b123-4ee0-8775-7bf81abe9658\") " pod="kuadrant-system/authorino-59c974bd48-jr25v"
Sep 16 13:50:45.369888 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.369863    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jr25v"
Sep 16 13:50:45.643590 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.643520    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jr25v"]
Sep 16 13:50:45.646879 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:50:45.646854    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda8cfc6c_b123_4ee0_8775_7bf81abe9658.slice/crio-836bb2f9c8341e00dc6a2737206b6bd369bf779226246f98a4eb38d858a20617 WatchSource:0}: Error finding container 836bb2f9c8341e00dc6a2737206b6bd369bf779226246f98a4eb38d858a20617: Status 404 returned error can't find the container with id 836bb2f9c8341e00dc6a2737206b6bd369bf779226246f98a4eb38d858a20617
Sep 16 13:50:45.871670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:45.871638    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jr25v" event={"ID":"da8cfc6c-b123-4ee0-8775-7bf81abe9658","Type":"ContainerStarted","Data":"836bb2f9c8341e00dc6a2737206b6bd369bf779226246f98a4eb38d858a20617"}
Sep 16 13:50:47.882935 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:47.882894    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jr25v" event={"ID":"da8cfc6c-b123-4ee0-8775-7bf81abe9658","Type":"ContainerStarted","Data":"85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e"}
Sep 16 13:50:47.900036 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:50:47.899983    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-jr25v" podStartSLOduration=1.16973306 podStartE2EDuration="2.899967143s" totalImagesPullingTime="1.730234083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:47.898722541 +0000 UTC m=+470.526203917" watchObservedRunningTime="2026-09-16 13:50:47.899967143 +0000 UTC m=+470.527448517"
Sep 16 13:51:19.769931 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:19.769894    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jr25v"]
Sep 16 13:51:19.770436 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:19.770060    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-jr25v" podUID="da8cfc6c-b123-4ee0-8775-7bf81abe9658" containerName="authorino" containerID="cri-o://85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e" gracePeriod=30
Sep 16 13:51:20.012693 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.012667    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jr25v"
Sep 16 13:51:20.019006 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.018985    2548 generic.go:356] "Generic (PLEG): container finished" podID="da8cfc6c-b123-4ee0-8775-7bf81abe9658" containerID="85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e" exitCode=0
Sep 16 13:51:20.019083 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.019022    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jr25v"
Sep 16 13:51:20.019083 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.019032    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jr25v" event={"ID":"da8cfc6c-b123-4ee0-8775-7bf81abe9658","Type":"ContainerDied","Data":"85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e"}
Sep 16 13:51:20.019083 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.019070    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jr25v" event={"ID":"da8cfc6c-b123-4ee0-8775-7bf81abe9658","Type":"ContainerDied","Data":"836bb2f9c8341e00dc6a2737206b6bd369bf779226246f98a4eb38d858a20617"}
Sep 16 13:51:20.019219 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.019087    2548 scope.go:117] "RemoveContainer" containerID="85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e"
Sep 16 13:51:20.026474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.026456    2548 scope.go:117] "RemoveContainer" containerID="85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e"
Sep 16 13:51:20.026711 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:51:20.026697    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e\": container with ID starting with 85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e not found: ID does not exist" containerID="85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e"
Sep 16 13:51:20.026767 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.026718    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e"} err="failed to get container status \"85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e\": rpc error: code = NotFound desc = could not find container \"85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e\": container with ID starting with 85c49d59d34cf9404e01f4be537382e6312fc5cc6991e80e452670f355fb6e4e not found: ID does not exist"
Sep 16 13:51:20.065418 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.065390    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkgdv\" (UniqueName: \"kubernetes.io/projected/da8cfc6c-b123-4ee0-8775-7bf81abe9658-kube-api-access-pkgdv\") pod \"da8cfc6c-b123-4ee0-8775-7bf81abe9658\" (UID: \"da8cfc6c-b123-4ee0-8775-7bf81abe9658\") "
Sep 16 13:51:20.067368 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.067342    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8cfc6c-b123-4ee0-8775-7bf81abe9658-kube-api-access-pkgdv" (OuterVolumeSpecName: "kube-api-access-pkgdv") pod "da8cfc6c-b123-4ee0-8775-7bf81abe9658" (UID: "da8cfc6c-b123-4ee0-8775-7bf81abe9658"). InnerVolumeSpecName "kube-api-access-pkgdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:51:20.166678 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.166651    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkgdv\" (UniqueName: \"kubernetes.io/projected/da8cfc6c-b123-4ee0-8775-7bf81abe9658-kube-api-access-pkgdv\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:51:20.347197 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.347168    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jr25v"]
Sep 16 13:51:20.359413 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:20.359373    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jr25v"]
Sep 16 13:51:21.969055 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:51:21.969026    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8cfc6c-b123-4ee0-8775-7bf81abe9658" path="/var/lib/kubelet/pods/da8cfc6c-b123-4ee0-8775-7bf81abe9658/volumes"
Sep 16 13:52:03.168636 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.168601    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"]
Sep 16 13:52:03.169124 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.168961    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8cfc6c-b123-4ee0-8775-7bf81abe9658" containerName="authorino"
Sep 16 13:52:03.169124 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.168976    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8cfc6c-b123-4ee0-8775-7bf81abe9658" containerName="authorino"
Sep 16 13:52:03.169124 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.169037    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8cfc6c-b123-4ee0-8775-7bf81abe9658" containerName="authorino"
Sep 16 13:52:03.173087 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.173067    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:03.177051 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.177016    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 13:52:03.177173 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.177054    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tffrm\""
Sep 16 13:52:03.187125 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.187102    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"]
Sep 16 13:52:03.224938 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.224908    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmxx\" (UniqueName: \"kubernetes.io/projected/01a23ec2-ae3a-4d16-95b2-8ed7ede19084-kube-api-access-bbmxx\") pod \"payload-pre-processing-59fbc47bf8-wzccf\" (UID: \"01a23ec2-ae3a-4d16-95b2-8ed7ede19084\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:03.225076 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.224944    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a23ec2-ae3a-4d16-95b2-8ed7ede19084-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wzccf\" (UID: \"01a23ec2-ae3a-4d16-95b2-8ed7ede19084\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:03.326160 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.326126    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a23ec2-ae3a-4d16-95b2-8ed7ede19084-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wzccf\" (UID: \"01a23ec2-ae3a-4d16-95b2-8ed7ede19084\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:03.326351 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.326223    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmxx\" (UniqueName: \"kubernetes.io/projected/01a23ec2-ae3a-4d16-95b2-8ed7ede19084-kube-api-access-bbmxx\") pod \"payload-pre-processing-59fbc47bf8-wzccf\" (UID: \"01a23ec2-ae3a-4d16-95b2-8ed7ede19084\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:03.326899 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.326878    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a23ec2-ae3a-4d16-95b2-8ed7ede19084-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wzccf\" (UID: \"01a23ec2-ae3a-4d16-95b2-8ed7ede19084\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:03.335872 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.335849    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmxx\" (UniqueName: \"kubernetes.io/projected/01a23ec2-ae3a-4d16-95b2-8ed7ede19084-kube-api-access-bbmxx\") pod \"payload-pre-processing-59fbc47bf8-wzccf\" (UID: \"01a23ec2-ae3a-4d16-95b2-8ed7ede19084\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:03.483710 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.483625    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:03.756104 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:03.756079    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"]
Sep 16 13:52:03.758431 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:52:03.758402    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a23ec2_ae3a_4d16_95b2_8ed7ede19084.slice/crio-3b2344a19617e25c4477ee61c26c52ebba17332acddb34ef97594ca0497d67d5 WatchSource:0}: Error finding container 3b2344a19617e25c4477ee61c26c52ebba17332acddb34ef97594ca0497d67d5: Status 404 returned error can't find the container with id 3b2344a19617e25c4477ee61c26c52ebba17332acddb34ef97594ca0497d67d5
Sep 16 13:52:04.207315 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:04.207275    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf" event={"ID":"01a23ec2-ae3a-4d16-95b2-8ed7ede19084","Type":"ContainerStarted","Data":"3b2344a19617e25c4477ee61c26c52ebba17332acddb34ef97594ca0497d67d5"}
Sep 16 13:52:07.228044 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:07.227981    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf" event={"ID":"01a23ec2-ae3a-4d16-95b2-8ed7ede19084","Type":"ContainerStarted","Data":"246499002fdddbe367759fdb0093e06af366998bea5dcfc12de518f896238699"}
Sep 16 13:52:07.228493 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:07.228144    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:07.244398 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:07.244344    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf" podStartSLOduration=1.55218678 podStartE2EDuration="4.244327596s" totalImagesPullingTime="2.692140816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:07.244220367 +0000 UTC m=+549.871701742" watchObservedRunningTime="2026-09-16 13:52:07.244327596 +0000 UTC m=+549.871808971"
Sep 16 13:52:13.256990 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:13.256958    2548 generic.go:356] "Generic (PLEG): container finished" podID="1ce50390-e6e1-49fe-9e01-8e74baea14ac" containerID="1600ad8b43cb196ae96804501d250b2d672e8641a5c58d127ddf79f4ea6d8303" exitCode=0
Sep 16 13:52:13.257433 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:13.257029    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" event={"ID":"1ce50390-e6e1-49fe-9e01-8e74baea14ac","Type":"ContainerDied","Data":"1600ad8b43cb196ae96804501d250b2d672e8641a5c58d127ddf79f4ea6d8303"}
Sep 16 13:52:13.257433 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:13.257420    2548 scope.go:117] "RemoveContainer" containerID="1600ad8b43cb196ae96804501d250b2d672e8641a5c58d127ddf79f4ea6d8303"
Sep 16 13:52:13.258670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:13.258635    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:52:13.258781 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:13.258722    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:52:13.258781 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:13.258768    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:52:14.263497 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:14.263461    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" event={"ID":"1ce50390-e6e1-49fe-9e01-8e74baea14ac","Type":"ContainerStarted","Data":"1bf62f2587aa703e9e739f52567ce157c83907689dc35944e578e43746880723"}
Sep 16 13:52:14.773131 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:14.773090    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:52:14.774124 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:14.774077    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-l5hkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 13:52:14.774232 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:14.774156    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" podUID="1ce50390-e6e1-49fe-9e01-8e74baea14ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 13:52:15.773502 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:15.773464    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-l5hkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 13:52:15.773891 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:15.773530    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" podUID="1ce50390-e6e1-49fe-9e01-8e74baea14ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 13:52:16.773398 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:16.773362    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-l5hkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 13:52:16.773557 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:16.773416    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" podUID="1ce50390-e6e1-49fe-9e01-8e74baea14ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 13:52:17.772878 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:17.772845    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-l5hkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 13:52:17.773036 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:17.772907    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" podUID="1ce50390-e6e1-49fe-9e01-8e74baea14ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 13:52:17.808563 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:17.808512    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:52:18.234707 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:18.234677    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wzccf"
Sep 16 13:52:18.772686 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:18.772652    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-l5hkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 13:52:18.772861 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:18.772706    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn" podUID="1ce50390-e6e1-49fe-9e01-8e74baea14ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 13:52:19.776850 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:19.776815    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:52:19.777871 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:19.777854    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-l5hkn"
Sep 16 13:52:37.416453 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.416420    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"]
Sep 16 13:52:37.420067 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.420047    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.422646 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.422624    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 13:52:37.423544 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.423526    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 16 13:52:37.423635 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.423539    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g67b7\""
Sep 16 13:52:37.423635 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.423542    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 13:52:37.430414 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.430392    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"]
Sep 16 13:52:37.535633 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.535604    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.535633 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.535639    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.535891 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.535661    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmvfw\" (UniqueName: \"kubernetes.io/projected/2f689721-b83b-4bb5-b959-997b2cdd0223-kube-api-access-vmvfw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.535891 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.535753    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.535891 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.535797    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.535891 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.535818    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f689721-b83b-4bb5-b959-997b2cdd0223-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.636733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.636692    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.636887 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.636747    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.636887 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.636776    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f689721-b83b-4bb5-b959-997b2cdd0223-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.636887 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.636840    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.636887 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.636870    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.637094 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.636896    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvfw\" (UniqueName: \"kubernetes.io/projected/2f689721-b83b-4bb5-b959-997b2cdd0223-kube-api-access-vmvfw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.637170 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.637139    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.637226 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.637170    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.637297 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.637271    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.639087 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.639070    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f689721-b83b-4bb5-b959-997b2cdd0223-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.639388 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.639372    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f689721-b83b-4bb5-b959-997b2cdd0223-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.645594 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.645567    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvfw\" (UniqueName: \"kubernetes.io/projected/2f689721-b83b-4bb5-b959-997b2cdd0223-kube-api-access-vmvfw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lqql\" (UID: \"2f689721-b83b-4bb5-b959-997b2cdd0223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:37.731163 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:37.731096    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:38.011133 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:38.011105    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"]
Sep 16 13:52:38.013749 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:52:38.013721    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f689721_b83b_4bb5_b959_997b2cdd0223.slice/crio-12011c496e1a55f0525a01d9a6a3ea6a4d0a67f2744ecfbbe41ecef0c17a526e WatchSource:0}: Error finding container 12011c496e1a55f0525a01d9a6a3ea6a4d0a67f2744ecfbbe41ecef0c17a526e: Status 404 returned error can't find the container with id 12011c496e1a55f0525a01d9a6a3ea6a4d0a67f2744ecfbbe41ecef0c17a526e
Sep 16 13:52:38.379105 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:38.379052    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql" event={"ID":"2f689721-b83b-4bb5-b959-997b2cdd0223","Type":"ContainerStarted","Data":"12011c496e1a55f0525a01d9a6a3ea6a4d0a67f2744ecfbbe41ecef0c17a526e"}
Sep 16 13:52:39.384720 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:39.384682    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql" event={"ID":"2f689721-b83b-4bb5-b959-997b2cdd0223","Type":"ContainerStarted","Data":"baa369489c5cece0a79dc574b88bd6c2b589bba6fe1c6ad1295dbabee30c15eb"}
Sep 16 13:52:39.403288 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:39.403232    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql" podStartSLOduration=1.3572871260000001 podStartE2EDuration="2.403213737s" totalImagesPullingTime="1.045926611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:39.402578515 +0000 UTC m=+582.030059902" watchObservedRunningTime="2026-09-16 13:52:39.403213737 +0000 UTC m=+582.030695112"
Sep 16 13:52:47.731880 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:47.731848    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:47.732331 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:47.732000    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:47.744109 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:47.744089    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:48.435385 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:48.435359    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lqql"
Sep 16 13:52:57.788665 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:57.788634    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:52:57.789558 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:52:57.789539    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:53:16.847766 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.847722    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"]
Sep 16 13:53:16.852750 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.852724    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:16.853908 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.853884    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"]
Sep 16 13:53:16.855492 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.855472    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 13:53:16.963526 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.963495    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a25fda7-a805-44e1-ad79-810b11c7a467-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:16.963526 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.963529    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:16.963726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.963558    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:16.963726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.963611    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:16.963726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.963691    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:16.963726 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:16.963711    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfnt\" (UniqueName: \"kubernetes.io/projected/6a25fda7-a805-44e1-ad79-810b11c7a467-kube-api-access-vnfnt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.064794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.064761    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a25fda7-a805-44e1-ad79-810b11c7a467-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.064794 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.064796    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.064990 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.064825    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.064990 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.064955    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.065100 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.065081    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.065142 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.065117    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfnt\" (UniqueName: \"kubernetes.io/projected/6a25fda7-a805-44e1-ad79-810b11c7a467-kube-api-access-vnfnt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.065226 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.065207    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.065320 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.065291    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.065474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.065454    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.067059 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.067038    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a25fda7-a805-44e1-ad79-810b11c7a467-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.067496 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.067476    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a25fda7-a805-44e1-ad79-810b11c7a467-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.074663 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.074640    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfnt\" (UniqueName: \"kubernetes.io/projected/6a25fda7-a805-44e1-ad79-810b11c7a467-kube-api-access-vnfnt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n\" (UID: \"6a25fda7-a805-44e1-ad79-810b11c7a467\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.163708 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.163671    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:17.437605 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.437580    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"]
Sep 16 13:53:17.440092 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:53:17.440058    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a25fda7_a805_44e1_ad79_810b11c7a467.slice/crio-2abbe5511aedf66ffc7b720b84a0f1ff0b8c803412a2d46ea11c30c1e111d91d WatchSource:0}: Error finding container 2abbe5511aedf66ffc7b720b84a0f1ff0b8c803412a2d46ea11c30c1e111d91d: Status 404 returned error can't find the container with id 2abbe5511aedf66ffc7b720b84a0f1ff0b8c803412a2d46ea11c30c1e111d91d
Sep 16 13:53:17.441434 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.441415    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:53:17.553199 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:17.553161    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n" event={"ID":"6a25fda7-a805-44e1-ad79-810b11c7a467","Type":"ContainerStarted","Data":"2abbe5511aedf66ffc7b720b84a0f1ff0b8c803412a2d46ea11c30c1e111d91d"}
Sep 16 13:53:18.559551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:18.559517    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n" event={"ID":"6a25fda7-a805-44e1-ad79-810b11c7a467","Type":"ContainerStarted","Data":"42615f0753998ea1ee52ffb63873cfb182a25b2396588052042729ce9fd3fbaa"}
Sep 16 13:53:18.579064 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:18.579012    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n" podStartSLOduration=2.377755853 podStartE2EDuration="2.578996092s" totalImagesPullingTime="201.240239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:18.577617681 +0000 UTC m=+621.205099055" watchObservedRunningTime="2026-09-16 13:53:18.578996092 +0000 UTC m=+621.206477466"
Sep 16 13:53:27.164946 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:27.164855    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:27.164946 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:27.164925    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:27.177034 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:27.177015    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:53:27.613913 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:53:27.613827    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n"
Sep 16 13:55:43.252333 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.252297    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"]
Sep 16 13:55:43.255817 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.255795    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.260344 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.260318    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-19203d-openshift-default-dockercfg-5rrwz\""
Sep 16 13:55:43.277568 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.277544    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"]
Sep 16 13:55:43.304037 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304007    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-token\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.304172 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304049    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-socket\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.304172 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304072    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istiod-ca-cert\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.304271 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304195    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-certs\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.304271 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304233    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-podinfo\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.304345 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304279    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mwv\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-kube-api-access-w9mwv\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.304345 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304316    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-envoy\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.304411 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304363    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-credential-socket\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.304411 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.304399    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-data\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.336867 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.336822    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"]
Sep 16 13:55:43.340144 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.340124    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.344098 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.344078    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b52a89-openshift-default-dockercfg-v8pcd\""
Sep 16 13:55:43.376388 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.376361    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"]
Sep 16 13:55:43.405887 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.405857    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mwv\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-kube-api-access-w9mwv\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406014 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.405901    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istiod-ca-cert\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406081 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406026    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-envoy\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406081 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406058    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-token\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406181 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406097    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-credential-socket\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406181 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406115    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvd4\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-kube-api-access-ncvd4\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406181 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406141    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-certs\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406181 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406163    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-data\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406397 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406188    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-data\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406397 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406261    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-socket\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406504 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406391    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-credential-socket\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406558 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406508    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-token\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406558 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406541    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-podinfo\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406658 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406551    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-data\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406658 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406569    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-credential-socket\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406658 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406572    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-socket\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406658 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406616    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istiod-ca-cert\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406658 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406645    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-envoy\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.406913 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406717    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-certs\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406913 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406743    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-podinfo\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.406913 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.406847    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-socket\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.407067 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.407040    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-certs\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.407361 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.407338    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istiod-ca-cert\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.408703 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.408677    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-envoy\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.409526 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.409507    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-podinfo\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.424263 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.423718    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mwv\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-kube-api-access-w9mwv\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.426428 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.425780    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-token\") pod \"e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") " pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.507556 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.507467    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-podinfo\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.507556 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.507513    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-envoy\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.507785 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.507571    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istiod-ca-cert\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.507785 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.507600    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-token\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.507785 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.507640    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvd4\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-kube-api-access-ncvd4\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.507785 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.507679    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-certs\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.508011 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.507979    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-data\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.508072 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.508035    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-socket\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.508072 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.508063    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-credential-socket\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.508556 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.508533    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-data\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.508854 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.508834    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-certs\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.508981 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.508887    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istiod-ca-cert\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.509095 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.509069    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-credential-socket\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.509165 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.509024    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-socket\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.510665 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.510634    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-podinfo\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.510852 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.510831    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-envoy\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.519781 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.519746    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvd4\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-kube-api-access-ncvd4\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.520897 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.520875    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-token\") pod \"e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") " pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.568430 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.568397    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:43.650611 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.650582    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:43.852839 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.852812    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"]
Sep 16 13:55:43.854216 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:55:43.854194    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431d1ed2_916a_40fd_b8ea_a506bdf4a668.slice/crio-6d71dca0b32b7331f9200ab902c72d7a52b2ab89b1bbac78297ac5e97a7cbd5e WatchSource:0}: Error finding container 6d71dca0b32b7331f9200ab902c72d7a52b2ab89b1bbac78297ac5e97a7cbd5e: Status 404 returned error can't find the container with id 6d71dca0b32b7331f9200ab902c72d7a52b2ab89b1bbac78297ac5e97a7cbd5e
Sep 16 13:55:43.856328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.856296    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:55:43.856421 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.856358    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:55:43.856421 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.856386    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:55:43.939948 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.939917    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"]
Sep 16 13:55:43.943811 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:55:43.943783    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd085667e_8ce6_402b_9d65_9f3ebf25abbb.slice/crio-b9740b21c868f5646e74ae6a7189b1009622fb2e0733812ef3061e1190fae68c WatchSource:0}: Error finding container b9740b21c868f5646e74ae6a7189b1009622fb2e0733812ef3061e1190fae68c: Status 404 returned error can't find the container with id b9740b21c868f5646e74ae6a7189b1009622fb2e0733812ef3061e1190fae68c
Sep 16 13:55:43.946372 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.946335    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:55:43.946474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.946429    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:55:43.946474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:43.946469    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:55:44.194854 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.194813    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" event={"ID":"d085667e-8ce6-402b-9d65-9f3ebf25abbb","Type":"ContainerStarted","Data":"d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711"}
Sep 16 13:55:44.195021 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.194877    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" event={"ID":"d085667e-8ce6-402b-9d65-9f3ebf25abbb","Type":"ContainerStarted","Data":"b9740b21c868f5646e74ae6a7189b1009622fb2e0733812ef3061e1190fae68c"}
Sep 16 13:55:44.197146 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.197115    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" event={"ID":"431d1ed2-916a-40fd-b8ea-a506bdf4a668","Type":"ContainerStarted","Data":"5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9"}
Sep 16 13:55:44.197291 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.197179    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" event={"ID":"431d1ed2-916a-40fd-b8ea-a506bdf4a668","Type":"ContainerStarted","Data":"6d71dca0b32b7331f9200ab902c72d7a52b2ab89b1bbac78297ac5e97a7cbd5e"}
Sep 16 13:55:44.268940 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.268881    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podStartSLOduration=1.268861209 podStartE2EDuration="1.268861209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:44.23551838 +0000 UTC m=+766.862999757" watchObservedRunningTime="2026-09-16 13:55:44.268861209 +0000 UTC m=+766.896342585"
Sep 16 13:55:44.270508 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.270468    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podStartSLOduration=1.2704550989999999 podStartE2EDuration="1.270455099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:44.269295795 +0000 UTC m=+766.896777170" watchObservedRunningTime="2026-09-16 13:55:44.270455099 +0000 UTC m=+766.897936474"
Sep 16 13:55:44.568670 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.568587    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:44.573108 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.573086    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:44.651879 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:44.651855    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:45.202313 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:45.202286    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:45.203228 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:45.203211    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:55:45.655831 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:45.655804    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:46.206482 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:46.206450    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:46.207418 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:46.207395    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:55:48.713741 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.713704    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"]
Sep 16 13:55:48.717313 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.717295    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:48.719888 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.719860    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-19203d-dockercfg-qgdcb\""
Sep 16 13:55:48.719888 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.719883    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-19203d\""
Sep 16 13:55:48.729999 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.729978    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"]
Sep 16 13:55:48.839546 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.839507    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"]
Sep 16 13:55:48.843029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.842998    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:48.855849 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.855823    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"]
Sep 16 13:55:48.857533 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.857510    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/9d5b971f-e015-4bb7-9de6-71ef9381b294-kube-api-access-s6nhr\") pod \"payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff\" (UID: \"9d5b971f-e015-4bb7-9de6-71ef9381b294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:48.857635 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.857538    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b971f-e015-4bb7-9de6-71ef9381b294-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff\" (UID: \"9d5b971f-e015-4bb7-9de6-71ef9381b294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:48.958874 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.958842    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba6caa-592d-4511-91b3-74bfea7a75ad-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw\" (UID: \"87ba6caa-592d-4511-91b3-74bfea7a75ad\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:48.959028 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.958898    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zpzf\" (UniqueName: \"kubernetes.io/projected/87ba6caa-592d-4511-91b3-74bfea7a75ad-kube-api-access-6zpzf\") pod \"payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw\" (UID: \"87ba6caa-592d-4511-91b3-74bfea7a75ad\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:48.959028 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.958931    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/9d5b971f-e015-4bb7-9de6-71ef9381b294-kube-api-access-s6nhr\") pod \"payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff\" (UID: \"9d5b971f-e015-4bb7-9de6-71ef9381b294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:48.959028 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.958948    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b971f-e015-4bb7-9de6-71ef9381b294-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff\" (UID: \"9d5b971f-e015-4bb7-9de6-71ef9381b294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:48.959705 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.959686    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b971f-e015-4bb7-9de6-71ef9381b294-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff\" (UID: \"9d5b971f-e015-4bb7-9de6-71ef9381b294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:48.972309 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:48.972230    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/9d5b971f-e015-4bb7-9de6-71ef9381b294-kube-api-access-s6nhr\") pod \"payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff\" (UID: \"9d5b971f-e015-4bb7-9de6-71ef9381b294\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:49.028073 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:49.028052    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:49.060409 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:49.060382    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zpzf\" (UniqueName: \"kubernetes.io/projected/87ba6caa-592d-4511-91b3-74bfea7a75ad-kube-api-access-6zpzf\") pod \"payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw\" (UID: \"87ba6caa-592d-4511-91b3-74bfea7a75ad\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:49.060551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:49.060527    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba6caa-592d-4511-91b3-74bfea7a75ad-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw\" (UID: \"87ba6caa-592d-4511-91b3-74bfea7a75ad\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:49.061515 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:49.061494    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba6caa-592d-4511-91b3-74bfea7a75ad-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw\" (UID: \"87ba6caa-592d-4511-91b3-74bfea7a75ad\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:49.077559 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:49.077519    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zpzf\" (UniqueName: \"kubernetes.io/projected/87ba6caa-592d-4511-91b3-74bfea7a75ad-kube-api-access-6zpzf\") pod \"payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw\" (UID: \"87ba6caa-592d-4511-91b3-74bfea7a75ad\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:49.154711 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:49.154674    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:49.309310 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:49.309282    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"]
Sep 16 13:55:49.311479 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:55:49.311442    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5b971f_e015_4bb7_9de6_71ef9381b294.slice/crio-edbb52a4c9224b409b2d29e9ebd9adb8793884bc3bcd2923062cb8a61a1d4b1c WatchSource:0}: Error finding container edbb52a4c9224b409b2d29e9ebd9adb8793884bc3bcd2923062cb8a61a1d4b1c: Status 404 returned error can't find the container with id edbb52a4c9224b409b2d29e9ebd9adb8793884bc3bcd2923062cb8a61a1d4b1c
Sep 16 13:55:49.436564 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:49.436539    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"]
Sep 16 13:55:49.438102 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:55:49.438074    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ba6caa_592d_4511_91b3_74bfea7a75ad.slice/crio-bd95fc8aa0f37156a0566cbe796ee8441a3ec741f546deaf6583ec141d00155b WatchSource:0}: Error finding container bd95fc8aa0f37156a0566cbe796ee8441a3ec741f546deaf6583ec141d00155b: Status 404 returned error can't find the container with id bd95fc8aa0f37156a0566cbe796ee8441a3ec741f546deaf6583ec141d00155b
Sep 16 13:55:50.231657 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.231615    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw" event={"ID":"87ba6caa-592d-4511-91b3-74bfea7a75ad","Type":"ContainerStarted","Data":"e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab"}
Sep 16 13:55:50.232106 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.231676    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw" event={"ID":"87ba6caa-592d-4511-91b3-74bfea7a75ad","Type":"ContainerStarted","Data":"bd95fc8aa0f37156a0566cbe796ee8441a3ec741f546deaf6583ec141d00155b"}
Sep 16 13:55:50.232106 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.231708    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:55:50.233333 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.233297    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff" event={"ID":"9d5b971f-e015-4bb7-9de6-71ef9381b294","Type":"ContainerStarted","Data":"e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6"}
Sep 16 13:55:50.233435 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.233348    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff" event={"ID":"9d5b971f-e015-4bb7-9de6-71ef9381b294","Type":"ContainerStarted","Data":"edbb52a4c9224b409b2d29e9ebd9adb8793884bc3bcd2923062cb8a61a1d4b1c"}
Sep 16 13:55:50.233435 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.233376    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:55:50.256407 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.256367    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw" podStartSLOduration=2.256353857 podStartE2EDuration="2.256353857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:50.255619605 +0000 UTC m=+772.883100978" watchObservedRunningTime="2026-09-16 13:55:50.256353857 +0000 UTC m=+772.883835230"
Sep 16 13:55:50.277355 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.277311    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff" podStartSLOduration=2.277293492 podStartE2EDuration="2.277293492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:50.276934362 +0000 UTC m=+772.904415738" watchObservedRunningTime="2026-09-16 13:55:50.277293492 +0000 UTC m=+772.904774885"
Sep 16 13:55:50.352867 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.349897    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"]
Sep 16 13:55:50.355335 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.355311    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:50.358331 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.358311    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b52a89-dockercfg-mm7j2\""
Sep 16 13:55:50.358431 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.358314    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b52a89\""
Sep 16 13:55:50.369722 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.369699    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"]
Sep 16 13:55:50.472957 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.472925    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgmm\" (UniqueName: \"kubernetes.io/projected/727b3305-af44-4e6b-ad30-bb17cbf3def0-kube-api-access-4qgmm\") pod \"payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt\" (UID: \"727b3305-af44-4e6b-ad30-bb17cbf3def0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:50.473140 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.473062    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b3305-af44-4e6b-ad30-bb17cbf3def0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt\" (UID: \"727b3305-af44-4e6b-ad30-bb17cbf3def0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:50.574152 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.574068    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgmm\" (UniqueName: \"kubernetes.io/projected/727b3305-af44-4e6b-ad30-bb17cbf3def0-kube-api-access-4qgmm\") pod \"payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt\" (UID: \"727b3305-af44-4e6b-ad30-bb17cbf3def0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:50.574329 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.574177    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b3305-af44-4e6b-ad30-bb17cbf3def0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt\" (UID: \"727b3305-af44-4e6b-ad30-bb17cbf3def0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:50.575072 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.575050    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b3305-af44-4e6b-ad30-bb17cbf3def0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt\" (UID: \"727b3305-af44-4e6b-ad30-bb17cbf3def0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:50.587351 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.587322    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgmm\" (UniqueName: \"kubernetes.io/projected/727b3305-af44-4e6b-ad30-bb17cbf3def0-kube-api-access-4qgmm\") pod \"payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt\" (UID: \"727b3305-af44-4e6b-ad30-bb17cbf3def0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:50.665921 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.665580    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:50.952181 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:50.952159    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"]
Sep 16 13:55:50.953667 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:55:50.953621    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727b3305_af44_4e6b_ad30_bb17cbf3def0.slice/crio-237b9f725a0f937e6144d44a6441b27c1b57b616858027db4c23a8140d343e74 WatchSource:0}: Error finding container 237b9f725a0f937e6144d44a6441b27c1b57b616858027db4c23a8140d343e74: Status 404 returned error can't find the container with id 237b9f725a0f937e6144d44a6441b27c1b57b616858027db4c23a8140d343e74
Sep 16 13:55:51.240106 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:51.240014    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt" event={"ID":"727b3305-af44-4e6b-ad30-bb17cbf3def0","Type":"ContainerStarted","Data":"27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95"}
Sep 16 13:55:51.240106 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:51.240079    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt" event={"ID":"727b3305-af44-4e6b-ad30-bb17cbf3def0","Type":"ContainerStarted","Data":"237b9f725a0f937e6144d44a6441b27c1b57b616858027db4c23a8140d343e74"}
Sep 16 13:55:51.240592 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:51.240413    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:55:51.262201 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:51.262159    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt" podStartSLOduration=1.262145775 podStartE2EDuration="1.262145775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:51.2602382 +0000 UTC m=+773.887719575" watchObservedRunningTime="2026-09-16 13:55:51.262145775 +0000 UTC m=+773.889627148"
Sep 16 13:55:54.975718 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:54.975685    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"]
Sep 16 13:55:54.979589 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:54.979564    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:55:54.983395 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:54.983365    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-035c64-dockercfg-fhccl\""
Sep 16 13:55:54.983863 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:54.983833    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-035c64\""
Sep 16 13:55:55.005325 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.005288    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"]
Sep 16 13:55:55.126981 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.126762    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d82ddf-4e94-4aad-a48e-05703d94231c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg\" (UID: \"a8d82ddf-4e94-4aad-a48e-05703d94231c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:55:55.126981 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.126873    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjh8\" (UniqueName: \"kubernetes.io/projected/a8d82ddf-4e94-4aad-a48e-05703d94231c-kube-api-access-zjjh8\") pod \"payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg\" (UID: \"a8d82ddf-4e94-4aad-a48e-05703d94231c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:55:55.228850 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.228752    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d82ddf-4e94-4aad-a48e-05703d94231c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg\" (UID: \"a8d82ddf-4e94-4aad-a48e-05703d94231c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:55:55.229155 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.229134    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjh8\" (UniqueName: \"kubernetes.io/projected/a8d82ddf-4e94-4aad-a48e-05703d94231c-kube-api-access-zjjh8\") pod \"payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg\" (UID: \"a8d82ddf-4e94-4aad-a48e-05703d94231c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:55:55.230451 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.230427    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d82ddf-4e94-4aad-a48e-05703d94231c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg\" (UID: \"a8d82ddf-4e94-4aad-a48e-05703d94231c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:55:55.242637 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.242583    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjh8\" (UniqueName: \"kubernetes.io/projected/a8d82ddf-4e94-4aad-a48e-05703d94231c-kube-api-access-zjjh8\") pod \"payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg\" (UID: \"a8d82ddf-4e94-4aad-a48e-05703d94231c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:55:55.296270 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.295471    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:55:55.600324 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:55.598655    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"]
Sep 16 13:55:55.603822 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:55:55.603760    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d82ddf_4e94_4aad_a48e_05703d94231c.slice/crio-99354fc3b0e19cb74206d8b7c6af51338cc1d3805005fa95cb8a94ba01441476 WatchSource:0}: Error finding container 99354fc3b0e19cb74206d8b7c6af51338cc1d3805005fa95cb8a94ba01441476: Status 404 returned error can't find the container with id 99354fc3b0e19cb74206d8b7c6af51338cc1d3805005fa95cb8a94ba01441476
Sep 16 13:55:56.268723 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:56.268691    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg" event={"ID":"a8d82ddf-4e94-4aad-a48e-05703d94231c","Type":"ContainerStarted","Data":"ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0"}
Sep 16 13:55:56.269114 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:56.268741    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg" event={"ID":"a8d82ddf-4e94-4aad-a48e-05703d94231c","Type":"ContainerStarted","Data":"99354fc3b0e19cb74206d8b7c6af51338cc1d3805005fa95cb8a94ba01441476"}
Sep 16 13:55:56.269328 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:55:56.269311    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:56:00.975912 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:00.975835    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg" podStartSLOduration=6.975810238 podStartE2EDuration="6.975810238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:56.291981004 +0000 UTC m=+778.919462379" watchObservedRunningTime="2026-09-16 13:56:00.975810238 +0000 UTC m=+783.603291629"
Sep 16 13:56:00.977790 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:00.977765    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"]
Sep 16 13:56:00.982403 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:00.982380    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:00.985467 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:00.985440    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-617597\""
Sep 16 13:56:00.994901 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:00.994875    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"]
Sep 16 13:56:01.096706 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.096653    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f025ac4-29db-408d-b8e3-7e797e4227b6-plugins-config-volume\") pod \"payload-processing-e2e-autores-617597-85b67fd5ff-dscqh\" (UID: \"7f025ac4-29db-408d-b8e3-7e797e4227b6\") " pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:01.096900 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.096717    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgd2m\" (UniqueName: \"kubernetes.io/projected/7f025ac4-29db-408d-b8e3-7e797e4227b6-kube-api-access-cgd2m\") pod \"payload-processing-e2e-autores-617597-85b67fd5ff-dscqh\" (UID: \"7f025ac4-29db-408d-b8e3-7e797e4227b6\") " pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:01.197782 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.197734    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f025ac4-29db-408d-b8e3-7e797e4227b6-plugins-config-volume\") pod \"payload-processing-e2e-autores-617597-85b67fd5ff-dscqh\" (UID: \"7f025ac4-29db-408d-b8e3-7e797e4227b6\") " pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:01.197986 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.197794    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgd2m\" (UniqueName: \"kubernetes.io/projected/7f025ac4-29db-408d-b8e3-7e797e4227b6-kube-api-access-cgd2m\") pod \"payload-processing-e2e-autores-617597-85b67fd5ff-dscqh\" (UID: \"7f025ac4-29db-408d-b8e3-7e797e4227b6\") " pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:01.198655 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.198622    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f025ac4-29db-408d-b8e3-7e797e4227b6-plugins-config-volume\") pod \"payload-processing-e2e-autores-617597-85b67fd5ff-dscqh\" (UID: \"7f025ac4-29db-408d-b8e3-7e797e4227b6\") " pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:01.208657 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.208622    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgd2m\" (UniqueName: \"kubernetes.io/projected/7f025ac4-29db-408d-b8e3-7e797e4227b6-kube-api-access-cgd2m\") pod \"payload-processing-e2e-autores-617597-85b67fd5ff-dscqh\" (UID: \"7f025ac4-29db-408d-b8e3-7e797e4227b6\") " pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:01.242116 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.242027    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 13:56:01.242116 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.242104    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 13:56:01.244426 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.244390    2548 patch_prober.go:28] interesting pod/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 13:56:01.244566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.244454    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 13:56:01.300588 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.300553    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:01.600839 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.600381    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"]
Sep 16 13:56:01.602753 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:56:01.602719    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f025ac4_29db_408d_b8e3_7e797e4227b6.slice/crio-bda9030bf2f1c46b044f640094a41614cd45c6e91788db94525a82c40b78d0db WatchSource:0}: Error finding container bda9030bf2f1c46b044f640094a41614cd45c6e91788db94525a82c40b78d0db: Status 404 returned error can't find the container with id bda9030bf2f1c46b044f640094a41614cd45c6e91788db94525a82c40b78d0db
Sep 16 13:56:01.869850 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.869817    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"]
Sep 16 13:56:01.876821 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.876802    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:01.880473 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.880295    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-95743e-openshift-default-dockercfg-92d47\""
Sep 16 13:56:01.893102 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:01.892455    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"]
Sep 16 13:56:02.010237 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010197    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-credential-socket\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.010698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010279    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-socket\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.010698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010337    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-podinfo\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.010698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010371    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-certs\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.010698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010396    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istiod-ca-cert\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.010698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010422    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-token\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.010698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010484    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-data\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.010698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010563    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-envoy\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.010698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.010601    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7qg\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-kube-api-access-mw7qg\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112198 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112150    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-token\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112402 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112211    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-data\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112402 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112280    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-envoy\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112402 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112316    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7qg\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-kube-api-access-mw7qg\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112404    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-credential-socket\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112442    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-socket\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112495    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-podinfo\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112529    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-certs\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112554    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istiod-ca-cert\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112814 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112654    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-data\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.112875 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.112837    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-credential-socket\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.113646 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.113599    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istiod-ca-cert\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.113771 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.113668    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-socket\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.113853 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.113832    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-certs\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.115816 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.115790    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-envoy\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.115940 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.115915    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-podinfo\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.130048 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.129987    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7qg\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-kube-api-access-mw7qg\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.130500 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.130475    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-token\") pod \"e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") " pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.194901 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.194862    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:02.247661 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.247624    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 13:56:02.248418 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.248386    2548 patch_prober.go:28] interesting pod/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 13:56:02.248535 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.248445    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 13:56:02.321040 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.320968    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh" event={"ID":"7f025ac4-29db-408d-b8e3-7e797e4227b6","Type":"ContainerStarted","Data":"e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2"}
Sep 16 13:56:02.321040 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.321037    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh" event={"ID":"7f025ac4-29db-408d-b8e3-7e797e4227b6","Type":"ContainerStarted","Data":"bda9030bf2f1c46b044f640094a41614cd45c6e91788db94525a82c40b78d0db"}
Sep 16 13:56:02.321557 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.321501    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:02.345210 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.345153    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh" podStartSLOduration=2.345133512 podStartE2EDuration="2.345133512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:02.342450817 +0000 UTC m=+784.969932188" watchObservedRunningTime="2026-09-16 13:56:02.345133512 +0000 UTC m=+784.972614888"
Sep 16 13:56:02.505336 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.504627    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"]
Sep 16 13:56:02.506847 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:56:02.506811    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c97b100_dcfc_47e3_9df5_d720d2b3ac96.slice/crio-aeaa93c7227143740a4d9a10b9e377610b19a0c2bc835167d137d9df7ccbb1a2 WatchSource:0}: Error finding container aeaa93c7227143740a4d9a10b9e377610b19a0c2bc835167d137d9df7ccbb1a2: Status 404 returned error can't find the container with id aeaa93c7227143740a4d9a10b9e377610b19a0c2bc835167d137d9df7ccbb1a2
Sep 16 13:56:02.510283 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.509965    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:02.510283 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.510056    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:02.510283 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.510098    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:02.552283 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.552231    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"]
Sep 16 13:56:02.557678 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.557651    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:02.560322 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.560298    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1dcc64e3-dockercfg-psqgw\""
Sep 16 13:56:02.560445 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.560298    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1dcc64e3\""
Sep 16 13:56:02.571505 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.571481    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"]
Sep 16 13:56:02.718715 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.718672    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9dffda-1ace-4312-a3f0-ed70b008271c-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l\" (UID: \"4f9dffda-1ace-4312-a3f0-ed70b008271c\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:02.718882 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.718822    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hln8\" (UniqueName: \"kubernetes.io/projected/4f9dffda-1ace-4312-a3f0-ed70b008271c-kube-api-access-2hln8\") pod \"payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l\" (UID: \"4f9dffda-1ace-4312-a3f0-ed70b008271c\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:02.820733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.820332    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hln8\" (UniqueName: \"kubernetes.io/projected/4f9dffda-1ace-4312-a3f0-ed70b008271c-kube-api-access-2hln8\") pod \"payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l\" (UID: \"4f9dffda-1ace-4312-a3f0-ed70b008271c\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:02.820733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.820475    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9dffda-1ace-4312-a3f0-ed70b008271c-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l\" (UID: \"4f9dffda-1ace-4312-a3f0-ed70b008271c\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:02.821488 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.821460    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9dffda-1ace-4312-a3f0-ed70b008271c-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l\" (UID: \"4f9dffda-1ace-4312-a3f0-ed70b008271c\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:02.836430 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.836391    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hln8\" (UniqueName: \"kubernetes.io/projected/4f9dffda-1ace-4312-a3f0-ed70b008271c-kube-api-access-2hln8\") pod \"payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l\" (UID: \"4f9dffda-1ace-4312-a3f0-ed70b008271c\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:02.880594 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:02.880555    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:03.171363 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:03.171337    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"]
Sep 16 13:56:03.173197 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:56:03.173166    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9dffda_1ace_4312_a3f0_ed70b008271c.slice/crio-bf3039f059f4aacf4876912a8b78acd9b64778f6a37073110b05083f1e0af99e WatchSource:0}: Error finding container bf3039f059f4aacf4876912a8b78acd9b64778f6a37073110b05083f1e0af99e: Status 404 returned error can't find the container with id bf3039f059f4aacf4876912a8b78acd9b64778f6a37073110b05083f1e0af99e
Sep 16 13:56:03.330606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:03.330571    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" event={"ID":"8c97b100-dcfc-47e3-9df5-d720d2b3ac96","Type":"ContainerStarted","Data":"0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61"}
Sep 16 13:56:03.330807 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:03.330635    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" event={"ID":"8c97b100-dcfc-47e3-9df5-d720d2b3ac96","Type":"ContainerStarted","Data":"aeaa93c7227143740a4d9a10b9e377610b19a0c2bc835167d137d9df7ccbb1a2"}
Sep 16 13:56:03.333172 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:03.333142    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l" event={"ID":"4f9dffda-1ace-4312-a3f0-ed70b008271c","Type":"ContainerStarted","Data":"afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773"}
Sep 16 13:56:03.333280 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:03.333201    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l" event={"ID":"4f9dffda-1ace-4312-a3f0-ed70b008271c","Type":"ContainerStarted","Data":"bf3039f059f4aacf4876912a8b78acd9b64778f6a37073110b05083f1e0af99e"}
Sep 16 13:56:03.363760 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:03.363700    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" podStartSLOduration=2.36367587 podStartE2EDuration="2.36367587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:03.360234884 +0000 UTC m=+785.987716270" watchObservedRunningTime="2026-09-16 13:56:03.36367587 +0000 UTC m=+785.991157245"
Sep 16 13:56:03.398405 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:03.396857    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l" podStartSLOduration=1.396836457 podStartE2EDuration="1.396836457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:03.395742847 +0000 UTC m=+786.023224243" watchObservedRunningTime="2026-09-16 13:56:03.396836457 +0000 UTC m=+786.024317832"
Sep 16 13:56:04.195792 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.195751    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:04.200935 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.200913    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:04.340070 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.340038    2548 generic.go:356] "Generic (PLEG): container finished" podID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerID="d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711" exitCode=0
Sep 16 13:56:04.340234 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.340114    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" event={"ID":"d085667e-8ce6-402b-9d65-9f3ebf25abbb","Type":"ContainerDied","Data":"d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711"}
Sep 16 13:56:04.340587 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.340560    2548 scope.go:117] "RemoveContainer" containerID="d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711"
Sep 16 13:56:04.341727 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.341687    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:04.341834 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.341770    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:04.341834 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.341795    2548 generic.go:356] "Generic (PLEG): container finished" podID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerID="5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9" exitCode=0
Sep 16 13:56:04.341834 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.341806    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:04.341987 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.341876    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" event={"ID":"431d1ed2-916a-40fd-b8ea-a506bdf4a668","Type":"ContainerDied","Data":"5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9"}
Sep 16 13:56:04.342365 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.342348    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:04.342441 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.342390    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:04.342568 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.342553    2548 scope.go:117] "RemoveContainer" containerID="5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9"
Sep 16 13:56:04.343579 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.343558    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:04.343656 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.343562    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:04.343656 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.343640    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:04.343738 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:04.343677    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:05.350718 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:05.350687    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" event={"ID":"d085667e-8ce6-402b-9d65-9f3ebf25abbb","Type":"ContainerStarted","Data":"c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd"}
Sep 16 13:56:05.352761 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:05.352733    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" event={"ID":"431d1ed2-916a-40fd-b8ea-a506bdf4a668","Type":"ContainerStarted","Data":"f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e"}
Sep 16 13:56:05.569157 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:05.569126    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:56:05.569402 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:05.569379    2548 patch_prober.go:28] interesting pod/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 13:56:05.569469 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:05.569451    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 13:56:05.650984 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:05.650956    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:56:05.651277 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:05.651158    2548 patch_prober.go:28] interesting pod/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:56:05.651277 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:05.651200    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 13:56:06.569611 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:06.569576    2548 patch_prober.go:28] interesting pod/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 13:56:06.570019 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:06.569630    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 13:56:06.653660 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:06.653629    2548 patch_prober.go:28] interesting pod/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:56:06.654327 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:06.654297    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 13:56:07.569309 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:07.569239    2548 patch_prober.go:28] interesting pod/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 13:56:07.569542 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:07.569339    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 13:56:07.652149 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:07.652109    2548 patch_prober.go:28] interesting pod/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:56:07.652528 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:07.652170    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 13:56:08.284483 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:08.284441    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 13:56:08.569510 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:08.569414    2548 patch_prober.go:28] interesting pod/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 13:56:08.569659 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:08.569510    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 13:56:08.651434 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:08.651397    2548 patch_prober.go:28] interesting pod/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:56:08.651595 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:08.651456    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 13:56:09.569267 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:09.569217    2548 patch_prober.go:28] interesting pod/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Sep 16 13:56:09.569644 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:09.569317    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Sep 16 13:56:09.651548 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:09.651512    2548 patch_prober.go:28] interesting pod/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:56:09.651712 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:09.651601    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 13:56:10.628324 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:10.628287    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:56:10.628957 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:10.628819    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:56:10.630107 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:10.630088    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 13:56:10.651942 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:10.651911    2548 patch_prober.go:28] interesting pod/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 16 13:56:10.652053 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:10.651956    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 16 13:56:11.655170 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:11.655141    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:56:11.655540 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:11.655436    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:56:11.656125 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:11.656107    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 13:56:13.341481 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:13.336052    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:56:15.355394 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:15.355365    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:28.479718 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:28.479680    2548 generic.go:356] "Generic (PLEG): container finished" podID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerID="0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61" exitCode=0
Sep 16 13:56:28.480130 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:28.479756    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" event={"ID":"8c97b100-dcfc-47e3-9df5-d720d2b3ac96","Type":"ContainerDied","Data":"0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61"}
Sep 16 13:56:28.480225 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:28.480208    2548 scope.go:117] "RemoveContainer" containerID="0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61"
Sep 16 13:56:28.481547 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:28.481519    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:28.481624 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:28.481587    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:28.481624 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:28.481619    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:56:29.487995 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:29.487963    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" event={"ID":"8c97b100-dcfc-47e3-9df5-d720d2b3ac96","Type":"ContainerStarted","Data":"6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8"}
Sep 16 13:56:30.197470 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:30.197437    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:30.197932 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:30.197904    2548 patch_prober.go:28] interesting pod/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 13:56:30.198162 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:30.198122    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 13:56:31.196682 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:31.196219    2548 patch_prober.go:28] interesting pod/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 13:56:31.196682 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:31.196300    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 13:56:32.195766 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:32.195725    2548 patch_prober.go:28] interesting pod/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 13:56:32.196509 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:32.196047    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 13:56:33.195835 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:33.195782    2548 patch_prober.go:28] interesting pod/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 13:56:33.196349 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:33.195859    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 13:56:34.196292 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:34.196230    2548 patch_prober.go:28] interesting pod/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 13:56:34.196886 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:34.196854    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 13:56:34.342366 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:34.342329    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:34.574740 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:34.574660    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"]
Sep 16 13:56:34.575004 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:34.574871    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l" podUID="4f9dffda-1ace-4312-a3f0-ed70b008271c" containerName="payload-processing" containerID="cri-o://afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773" gracePeriod=30
Sep 16 13:56:34.861414 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:34.861389    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:35.052577 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.052544    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hln8\" (UniqueName: \"kubernetes.io/projected/4f9dffda-1ace-4312-a3f0-ed70b008271c-kube-api-access-2hln8\") pod \"4f9dffda-1ace-4312-a3f0-ed70b008271c\" (UID: \"4f9dffda-1ace-4312-a3f0-ed70b008271c\") "
Sep 16 13:56:35.052745 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.052687    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9dffda-1ace-4312-a3f0-ed70b008271c-plugins-config-volume\") pod \"4f9dffda-1ace-4312-a3f0-ed70b008271c\" (UID: \"4f9dffda-1ace-4312-a3f0-ed70b008271c\") "
Sep 16 13:56:35.053007 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.052984    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9dffda-1ace-4312-a3f0-ed70b008271c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f9dffda-1ace-4312-a3f0-ed70b008271c" (UID: "4f9dffda-1ace-4312-a3f0-ed70b008271c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:56:35.054741 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.054718    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9dffda-1ace-4312-a3f0-ed70b008271c-kube-api-access-2hln8" (OuterVolumeSpecName: "kube-api-access-2hln8") pod "4f9dffda-1ace-4312-a3f0-ed70b008271c" (UID: "4f9dffda-1ace-4312-a3f0-ed70b008271c"). InnerVolumeSpecName "kube-api-access-2hln8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:56:35.153566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.153536    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hln8\" (UniqueName: \"kubernetes.io/projected/4f9dffda-1ace-4312-a3f0-ed70b008271c-kube-api-access-2hln8\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:56:35.153566 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.153561    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f9dffda-1ace-4312-a3f0-ed70b008271c-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:56:35.196365 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.195879    2548 patch_prober.go:28] interesting pod/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 13:56:35.196365 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.195947    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 13:56:35.533310 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.533204    2548 generic.go:356] "Generic (PLEG): container finished" podID="4f9dffda-1ace-4312-a3f0-ed70b008271c" containerID="afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773" exitCode=0
Sep 16 13:56:35.533310 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.533266    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"
Sep 16 13:56:35.533310 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.533295    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l" event={"ID":"4f9dffda-1ace-4312-a3f0-ed70b008271c","Type":"ContainerDied","Data":"afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773"}
Sep 16 13:56:35.533518 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.533352    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l" event={"ID":"4f9dffda-1ace-4312-a3f0-ed70b008271c","Type":"ContainerDied","Data":"bf3039f059f4aacf4876912a8b78acd9b64778f6a37073110b05083f1e0af99e"}
Sep 16 13:56:35.533518 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.533374    2548 scope.go:117] "RemoveContainer" containerID="afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773"
Sep 16 13:56:35.542087 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.542067    2548 scope.go:117] "RemoveContainer" containerID="afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773"
Sep 16 13:56:35.542363 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:56:35.542344    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773\": container with ID starting with afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773 not found: ID does not exist" containerID="afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773"
Sep 16 13:56:35.542412 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.542372    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773"} err="failed to get container status \"afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773\": rpc error: code = NotFound desc = could not find container \"afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773\": container with ID starting with afca7233c49469b61bbae0fd86be990931575afeb948804871c9e83f94bfa773 not found: ID does not exist"
Sep 16 13:56:35.566434 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.566410    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"]
Sep 16 13:56:35.572237 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.572210    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcc64e3-75fc99dbdf-kbk7l"]
Sep 16 13:56:35.971622 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:35.971585    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f9dffda-1ace-4312-a3f0-ed70b008271c" path="/var/lib/kubelet/pods/4f9dffda-1ace-4312-a3f0-ed70b008271c/volumes"
Sep 16 13:56:36.200346 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:36.200318    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:56:36.201435 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:56:36.201419    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 13:57:03.757217 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.757181    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"]
Sep 16 13:57:03.757674 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.757598    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f9dffda-1ace-4312-a3f0-ed70b008271c" containerName="payload-processing"
Sep 16 13:57:03.757674 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.757611    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f9dffda-1ace-4312-a3f0-ed70b008271c" containerName="payload-processing"
Sep 16 13:57:03.757749 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.757699    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f9dffda-1ace-4312-a3f0-ed70b008271c" containerName="payload-processing"
Sep 16 13:57:03.760606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.760587    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"
Sep 16 13:57:03.763524 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.763506    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:57:03.763628 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.763528    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 13:57:03.771240 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.771221    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"]
Sep 16 13:57:03.921980 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:03.921943    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4626fd-736c-438d-9e90-2e17b521a13c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4\" (UID: \"da4626fd-736c-438d-9e90-2e17b521a13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"
Sep 16 13:57:04.022475 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:04.022384    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4626fd-736c-438d-9e90-2e17b521a13c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4\" (UID: \"da4626fd-736c-438d-9e90-2e17b521a13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"
Sep 16 13:57:04.023029 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:04.023009    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4626fd-736c-438d-9e90-2e17b521a13c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4\" (UID: \"da4626fd-736c-438d-9e90-2e17b521a13c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"
Sep 16 13:57:04.071321 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:04.071294    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"
Sep 16 13:57:04.356323 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:04.356272    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"]
Sep 16 13:57:04.358713 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:57:04.358687    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4626fd_736c_438d_9e90_2e17b521a13c.slice/crio-e3f5bcae42c1f7aad3a29e8742d85aa4f6bbd8d643d0afbf98788cbf429df36f WatchSource:0}: Error finding container e3f5bcae42c1f7aad3a29e8742d85aa4f6bbd8d643d0afbf98788cbf429df36f: Status 404 returned error can't find the container with id e3f5bcae42c1f7aad3a29e8742d85aa4f6bbd8d643d0afbf98788cbf429df36f
Sep 16 13:57:04.689891 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:04.689857    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4" event={"ID":"da4626fd-736c-438d-9e90-2e17b521a13c","Type":"ContainerStarted","Data":"e3f5bcae42c1f7aad3a29e8742d85aa4f6bbd8d643d0afbf98788cbf429df36f"}
Sep 16 13:57:07.706506 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:07.706468    2548 generic.go:356] "Generic (PLEG): container finished" podID="da4626fd-736c-438d-9e90-2e17b521a13c" containerID="c1335faa61c11c6dcf17b79612ce66e250ae5c2945603763871b66a7ca096bb4" exitCode=0
Sep 16 13:57:07.706890 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:07.706524    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4" event={"ID":"da4626fd-736c-438d-9e90-2e17b521a13c","Type":"ContainerDied","Data":"c1335faa61c11c6dcf17b79612ce66e250ae5c2945603763871b66a7ca096bb4"}
Sep 16 13:57:08.844058 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:08.844032    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"
Sep 16 13:57:08.974398 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:08.974316    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4626fd-736c-438d-9e90-2e17b521a13c-maas-api-service-ca\") pod \"da4626fd-736c-438d-9e90-2e17b521a13c\" (UID: \"da4626fd-736c-438d-9e90-2e17b521a13c\") "
Sep 16 13:57:08.974707 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:08.974681    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4626fd-736c-438d-9e90-2e17b521a13c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da4626fd-736c-438d-9e90-2e17b521a13c" (UID: "da4626fd-736c-438d-9e90-2e17b521a13c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:57:09.075293 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:09.075267    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4626fd-736c-438d-9e90-2e17b521a13c-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:57:09.718263 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:09.718232    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"
Sep 16 13:57:09.718420 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:09.718278    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4" event={"ID":"da4626fd-736c-438d-9e90-2e17b521a13c","Type":"ContainerDied","Data":"e3f5bcae42c1f7aad3a29e8742d85aa4f6bbd8d643d0afbf98788cbf429df36f"}
Sep 16 13:57:09.718420 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:09.718329    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f5bcae42c1f7aad3a29e8742d85aa4f6bbd8d643d0afbf98788cbf429df36f"
Sep 16 13:57:21.081443 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.081403    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"]
Sep 16 13:57:21.087223 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.087196    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ce782c2e-5rwr4"]
Sep 16 13:57:21.971749 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.971710    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da4626fd-736c-438d-9e90-2e17b521a13c" path="/var/lib/kubelet/pods/da4626fd-736c-438d-9e90-2e17b521a13c/volumes"
Sep 16 13:57:21.972012 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.971996    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"]
Sep 16 13:57:21.972658 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.972592    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4626fd-736c-438d-9e90-2e17b521a13c" containerName="revoke-keys"
Sep 16 13:57:21.972658 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.972627    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4626fd-736c-438d-9e90-2e17b521a13c" containerName="revoke-keys"
Sep 16 13:57:21.972881 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.972850    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4626fd-736c-438d-9e90-2e17b521a13c" containerName="revoke-keys"
Sep 16 13:57:21.981156 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.981133    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:21.983901 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.983877    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-38e8bf38-openshift-default-dockercfg-8mwsm\""
Sep 16 13:57:21.988390 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:21.988308    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"]
Sep 16 13:57:22.002911 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.002880    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-token\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.003046 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.002943    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-credential-socket\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.003046 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.002987    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-podinfo\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.003046 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.003037    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-data\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.003223 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.003071    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-socket\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.003223 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.003122    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istiod-ca-cert\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.003223 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.003152    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7kp\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-kube-api-access-qf7kp\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.003421 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.003228    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-envoy\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.003421 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.003283    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-certs\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.103879 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.103846    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-token\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104395 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.103899    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-credential-socket\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104395 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.103925    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-podinfo\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104395 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.103956    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-data\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104395 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.103980    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-socket\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104395 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.104002    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istiod-ca-cert\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104395 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.104031    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7kp\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-kube-api-access-qf7kp\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104721 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.104637    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-data\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104783 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.104756    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-envoy\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.104835 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.104818    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-certs\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.105136 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.105109    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-certs\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.105309 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.105205    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istiod-ca-cert\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.105467 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.105440    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-socket\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.105549 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.105472    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-credential-socket\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.106928 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.106903    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-envoy\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.107019 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.106952    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-podinfo\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.116075 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.116047    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-token\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.116474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.116452    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7kp\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-kube-api-access-qf7kp\") pod \"e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") " pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.295075 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.294998    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:22.576664 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.576638    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"]
Sep 16 13:57:22.578411 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:57:22.578388    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode924a7b6_0eb0_436f_a70b_e1433f0460ee.slice/crio-27828b180862c30b71933bc0838d6f335fec640fc41857ce437c0002be82ee58 WatchSource:0}: Error finding container 27828b180862c30b71933bc0838d6f335fec640fc41857ce437c0002be82ee58: Status 404 returned error can't find the container with id 27828b180862c30b71933bc0838d6f335fec640fc41857ce437c0002be82ee58
Sep 16 13:57:22.580747 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.580713    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:57:22.580834 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.580778    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:57:22.580834 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.580807    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:57:22.785305 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.785262    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" event={"ID":"e924a7b6-0eb0-436f-a70b-e1433f0460ee","Type":"ContainerStarted","Data":"591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480"}
Sep 16 13:57:22.785486 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.785334    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" event={"ID":"e924a7b6-0eb0-436f-a70b-e1433f0460ee","Type":"ContainerStarted","Data":"27828b180862c30b71933bc0838d6f335fec640fc41857ce437c0002be82ee58"}
Sep 16 13:57:22.814100 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:22.814052    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" podStartSLOduration=1.814036161 podStartE2EDuration="1.814036161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:22.813132819 +0000 UTC m=+865.440614214" watchObservedRunningTime="2026-09-16 13:57:22.814036161 +0000 UTC m=+865.441517535"
Sep 16 13:57:23.295958 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:23.295922    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:23.300928 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:23.300903    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:23.790863 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:23.790833    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:23.791659 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:23.791639    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:48.939918 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:48.939828    2548 generic.go:356] "Generic (PLEG): container finished" podID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerID="591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480" exitCode=0
Sep 16 13:57:48.940348 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:48.939908    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" event={"ID":"e924a7b6-0eb0-436f-a70b-e1433f0460ee","Type":"ContainerDied","Data":"591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480"}
Sep 16 13:57:48.940348 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:48.940321    2548 scope.go:117] "RemoveContainer" containerID="591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480"
Sep 16 13:57:48.941463 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:48.941438    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:57:48.941523 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:48.941502    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:57:48.941562 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:48.941533    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:57:49.947609 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:49.947571    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" event={"ID":"e924a7b6-0eb0-436f-a70b-e1433f0460ee","Type":"ContainerStarted","Data":"8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966"}
Sep 16 13:57:50.296089 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:50.295987    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:50.297286 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:50.297259    2548 patch_prober.go:28] interesting pod/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 16 13:57:50.297484 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:50.297458    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 16 13:57:51.296261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:51.295705    2548 patch_prober.go:28] interesting pod/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 16 13:57:51.296261 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:51.295841    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 16 13:57:52.295404 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:52.295361    2548 patch_prober.go:28] interesting pod/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 16 13:57:52.295684 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:52.295653    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 16 13:57:53.295552 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:53.295506    2548 patch_prober.go:28] interesting pod/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 16 13:57:53.296224 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:53.295610    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 16 13:57:53.791568 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:53.791533    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:54.296113 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:54.296069    2548 patch_prober.go:28] interesting pod/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 16 13:57:54.296609 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:54.296172    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 16 13:57:54.997552 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:54.997519    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"]
Sep 16 13:57:55.000378 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.000357    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"
Sep 16 13:57:55.003005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.002985    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:57:55.003122 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.003089    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 13:57:55.016733 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.016704    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"]
Sep 16 13:57:55.140168 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.140139    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a6af48-68e7-44bd-b445-3fea338ed711-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4\" (UID: \"36a6af48-68e7-44bd-b445-3fea338ed711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"
Sep 16 13:57:55.240931 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.240904    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a6af48-68e7-44bd-b445-3fea338ed711-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4\" (UID: \"36a6af48-68e7-44bd-b445-3fea338ed711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"
Sep 16 13:57:55.241559 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.241542    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a6af48-68e7-44bd-b445-3fea338ed711-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4\" (UID: \"36a6af48-68e7-44bd-b445-3fea338ed711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"
Sep 16 13:57:55.299479 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.299408    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:55.300552 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.300529    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:57:55.310923 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.310905    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"
Sep 16 13:57:55.585747 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.585723    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"]
Sep 16 13:57:55.588166 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:57:55.588138    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a6af48_68e7_44bd_b445_3fea338ed711.slice/crio-3de9a1d55f5c39403e917a07de32b9b1c804cdfa1767ce5347ef43b0cbe014fb WatchSource:0}: Error finding container 3de9a1d55f5c39403e917a07de32b9b1c804cdfa1767ce5347ef43b0cbe014fb: Status 404 returned error can't find the container with id 3de9a1d55f5c39403e917a07de32b9b1c804cdfa1767ce5347ef43b0cbe014fb
Sep 16 13:57:55.979306 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.979274    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4" event={"ID":"36a6af48-68e7-44bd-b445-3fea338ed711","Type":"ContainerStarted","Data":"1ef68645c67b0a1fe9e7c2c994781b92227384c2a2d3264100e7ba1c77124c8c"}
Sep 16 13:57:55.979481 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.979324    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4" event={"ID":"36a6af48-68e7-44bd-b445-3fea338ed711","Type":"ContainerStarted","Data":"3de9a1d55f5c39403e917a07de32b9b1c804cdfa1767ce5347ef43b0cbe014fb"}
Sep 16 13:57:55.996951 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:55.996903    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4" podStartSLOduration=1.996889748 podStartE2EDuration="1.996889748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:55.995327976 +0000 UTC m=+898.622809349" watchObservedRunningTime="2026-09-16 13:57:55.996889748 +0000 UTC m=+898.624371171"
Sep 16 13:57:56.986106 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:56.986071    2548 generic.go:356] "Generic (PLEG): container finished" podID="36a6af48-68e7-44bd-b445-3fea338ed711" containerID="1ef68645c67b0a1fe9e7c2c994781b92227384c2a2d3264100e7ba1c77124c8c" exitCode=0
Sep 16 13:57:56.986539 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:56.986159    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4" event={"ID":"36a6af48-68e7-44bd-b445-3fea338ed711","Type":"ContainerDied","Data":"1ef68645c67b0a1fe9e7c2c994781b92227384c2a2d3264100e7ba1c77124c8c"}
Sep 16 13:57:57.822698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:57.822670    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:57:57.826538 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:57.823558    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 13:57:58.124674 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:58.124649    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"
Sep 16 13:57:58.165643 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:58.165615    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a6af48-68e7-44bd-b445-3fea338ed711-maas-api-service-ca\") pod \"36a6af48-68e7-44bd-b445-3fea338ed711\" (UID: \"36a6af48-68e7-44bd-b445-3fea338ed711\") "
Sep 16 13:57:58.165925 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:58.165898    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a6af48-68e7-44bd-b445-3fea338ed711-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36a6af48-68e7-44bd-b445-3fea338ed711" (UID: "36a6af48-68e7-44bd-b445-3fea338ed711"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:57:58.266949 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:58.266923    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a6af48-68e7-44bd-b445-3fea338ed711-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:57:58.997005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:58.996969    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4" event={"ID":"36a6af48-68e7-44bd-b445-3fea338ed711","Type":"ContainerDied","Data":"3de9a1d55f5c39403e917a07de32b9b1c804cdfa1767ce5347ef43b0cbe014fb"}
Sep 16 13:57:58.997005 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:58.996994    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"
Sep 16 13:57:58.997214 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:57:58.997020    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3de9a1d55f5c39403e917a07de32b9b1c804cdfa1767ce5347ef43b0cbe014fb"
Sep 16 13:58:04.329159 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.329115    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"]
Sep 16 13:58:04.329771 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.329749    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a6af48-68e7-44bd-b445-3fea338ed711" containerName="revoke-keys"
Sep 16 13:58:04.329857 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.329772    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a6af48-68e7-44bd-b445-3fea338ed711" containerName="revoke-keys"
Sep 16 13:58:04.329911 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.329870    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a6af48-68e7-44bd-b445-3fea338ed711" containerName="revoke-keys"
Sep 16 13:58:04.331918 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.331898    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"
Sep 16 13:58:04.334843 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.334822    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:58:04.334955 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.334825    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 13:58:04.342689 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.342666    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"]
Sep 16 13:58:04.425125 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.425090    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-617597-c2xfm\" (UID: \"6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"
Sep 16 13:58:04.525635 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.525594    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-617597-c2xfm\" (UID: \"6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"
Sep 16 13:58:04.526181 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.526162    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-617597-c2xfm\" (UID: \"6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"
Sep 16 13:58:04.642508 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.642479    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"
Sep 16 13:58:04.917859 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:04.917833    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"]
Sep 16 13:58:04.919241 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:58:04.919212    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2c4d24_b889_4b8a_88bd_2f82e7f42bd7.slice/crio-9f41f6c4712d6eba312c11998e25050ee8f3544c7fe4e4972622fe958eaffac9 WatchSource:0}: Error finding container 9f41f6c4712d6eba312c11998e25050ee8f3544c7fe4e4972622fe958eaffac9: Status 404 returned error can't find the container with id 9f41f6c4712d6eba312c11998e25050ee8f3544c7fe4e4972622fe958eaffac9
Sep 16 13:58:05.026067 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:05.026040    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm" event={"ID":"6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7","Type":"ContainerStarted","Data":"9f41f6c4712d6eba312c11998e25050ee8f3544c7fe4e4972622fe958eaffac9"}
Sep 16 13:58:06.032799 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:06.032766    2548 generic.go:356] "Generic (PLEG): container finished" podID="6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7" containerID="9363a296cbd0fe6c3e5f1c59a8b29be29d4b1c1e2a380fbd7c143350ebc601bd" exitCode=0
Sep 16 13:58:06.033189 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:06.032850    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm" event={"ID":"6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7","Type":"ContainerDied","Data":"9363a296cbd0fe6c3e5f1c59a8b29be29d4b1c1e2a380fbd7c143350ebc601bd"}
Sep 16 13:58:07.172638 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:07.172607    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"
Sep 16 13:58:07.251345 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:07.251311    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7-maas-api-service-ca\") pod \"6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7\" (UID: \"6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7\") "
Sep 16 13:58:07.251635 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:07.251610    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7" (UID: "6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:58:07.251709 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:07.251692    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:08.043965 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:08.043939    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"
Sep 16 13:58:08.044125 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:08.043972    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm" event={"ID":"6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7","Type":"ContainerDied","Data":"9f41f6c4712d6eba312c11998e25050ee8f3544c7fe4e4972622fe958eaffac9"}
Sep 16 13:58:08.044125 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:08.044014    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f41f6c4712d6eba312c11998e25050ee8f3544c7fe4e4972622fe958eaffac9"
Sep 16 13:58:12.457801 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:12.457751    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"]
Sep 16 13:58:12.462119 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:12.462092    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-38e8bf38-knzf4"]
Sep 16 13:58:13.602186 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:13.602151    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"]
Sep 16 13:58:13.602614 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:13.602438    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy" containerID="cri-o://8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966" gracePeriod=30
Sep 16 13:58:13.969994 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:13.969952    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a6af48-68e7-44bd-b445-3fea338ed711" path="/var/lib/kubelet/pods/36a6af48-68e7-44bd-b445-3fea338ed711/volumes"
Sep 16 13:58:17.610481 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:17.610450    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"]
Sep 16 13:58:17.610932 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:17.610696    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh" podUID="7f025ac4-29db-408d-b8e3-7e797e4227b6" containerName="payload-processing" containerID="cri-o://e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2" gracePeriod=30
Sep 16 13:58:17.890228 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:17.890201    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:58:17.958627 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:17.958600    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f025ac4-29db-408d-b8e3-7e797e4227b6-plugins-config-volume\") pod \"7f025ac4-29db-408d-b8e3-7e797e4227b6\" (UID: \"7f025ac4-29db-408d-b8e3-7e797e4227b6\") "
Sep 16 13:58:17.958779 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:17.958698    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgd2m\" (UniqueName: \"kubernetes.io/projected/7f025ac4-29db-408d-b8e3-7e797e4227b6-kube-api-access-cgd2m\") pod \"7f025ac4-29db-408d-b8e3-7e797e4227b6\" (UID: \"7f025ac4-29db-408d-b8e3-7e797e4227b6\") "
Sep 16 13:58:17.958986 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:17.958962    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f025ac4-29db-408d-b8e3-7e797e4227b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f025ac4-29db-408d-b8e3-7e797e4227b6" (UID: "7f025ac4-29db-408d-b8e3-7e797e4227b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:58:17.960659 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:17.960637    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f025ac4-29db-408d-b8e3-7e797e4227b6-kube-api-access-cgd2m" (OuterVolumeSpecName: "kube-api-access-cgd2m") pod "7f025ac4-29db-408d-b8e3-7e797e4227b6" (UID: "7f025ac4-29db-408d-b8e3-7e797e4227b6"). InnerVolumeSpecName "kube-api-access-cgd2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:58:17.995832 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:17.995804    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"]
Sep 16 13:58:18.000149 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.000125    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-617597-c2xfm"]
Sep 16 13:58:18.059698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.059668    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgd2m\" (UniqueName: \"kubernetes.io/projected/7f025ac4-29db-408d-b8e3-7e797e4227b6-kube-api-access-cgd2m\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:18.059698 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.059695    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f025ac4-29db-408d-b8e3-7e797e4227b6-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:18.105340 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.105307    2548 generic.go:356] "Generic (PLEG): container finished" podID="7f025ac4-29db-408d-b8e3-7e797e4227b6" containerID="e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2" exitCode=0
Sep 16 13:58:18.105535 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.105369    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"
Sep 16 13:58:18.105535 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.105364    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh" event={"ID":"7f025ac4-29db-408d-b8e3-7e797e4227b6","Type":"ContainerDied","Data":"e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2"}
Sep 16 13:58:18.105535 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.105483    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh" event={"ID":"7f025ac4-29db-408d-b8e3-7e797e4227b6","Type":"ContainerDied","Data":"bda9030bf2f1c46b044f640094a41614cd45c6e91788db94525a82c40b78d0db"}
Sep 16 13:58:18.105535 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.105507    2548 scope.go:117] "RemoveContainer" containerID="e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2"
Sep 16 13:58:18.114295 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.114278    2548 scope.go:117] "RemoveContainer" containerID="e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2"
Sep 16 13:58:18.114555 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:58:18.114535    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2\": container with ID starting with e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2 not found: ID does not exist" containerID="e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2"
Sep 16 13:58:18.114615 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.114568    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2"} err="failed to get container status \"e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2\": rpc error: code = NotFound desc = could not find container \"e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2\": container with ID starting with e4c4d71a9234353e921b1d3fd1aabbcaabb2a1ded26ca4fd3e9ad72d9e7c89f2 not found: ID does not exist"
Sep 16 13:58:18.126669 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.126611    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"]
Sep 16 13:58:18.131750 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.131729    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-617597-85b67fd5ff-dscqh"]
Sep 16 13:58:18.850881 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.850859    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:58:18.969409 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969374    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-data\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969453    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-envoy\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969484    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-credential-socket\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969512    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-podinfo\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969537    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istiod-ca-cert\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969558    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-certs\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969598    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf7kp\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-kube-api-access-qf7kp\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969929 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969625    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-token\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969929 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969674    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-socket\") pod \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\" (UID: \"e924a7b6-0eb0-436f-a70b-e1433f0460ee\") "
Sep 16 13:58:18.969929 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969671    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:18.969929 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969787    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:18.970152 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.969990    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:18.970152 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.970010    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:58:18.970152 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.970120    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:18.970152 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.970142    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:18.970394 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.970154    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:18.970394 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.970201    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:18.971731 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.971707    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:58:18.971845 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.971761    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-kube-api-access-qf7kp" (OuterVolumeSpecName: "kube-api-access-qf7kp") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "kube-api-access-qf7kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:58:18.972069 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.972041    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:18.972069 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:18.972049    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "e924a7b6-0eb0-436f-a70b-e1433f0460ee" (UID: "e924a7b6-0eb0-436f-a70b-e1433f0460ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:58:19.071623 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.071589    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf7kp\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-kube-api-access-qf7kp\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:19.071623 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.071620    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:19.071623 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.071630    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:19.071864 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.071637    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:19.071864 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.071645    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e924a7b6-0eb0-436f-a70b-e1433f0460ee-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:19.071864 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.071653    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e924a7b6-0eb0-436f-a70b-e1433f0460ee-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:58:19.113201 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.113170    2548 generic.go:356] "Generic (PLEG): container finished" podID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerID="8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966" exitCode=0
Sep 16 13:58:19.113404 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.113222    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"
Sep 16 13:58:19.113404 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.113235    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" event={"ID":"e924a7b6-0eb0-436f-a70b-e1433f0460ee","Type":"ContainerDied","Data":"8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966"}
Sep 16 13:58:19.113404 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.113295    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf" event={"ID":"e924a7b6-0eb0-436f-a70b-e1433f0460ee","Type":"ContainerDied","Data":"27828b180862c30b71933bc0838d6f335fec640fc41857ce437c0002be82ee58"}
Sep 16 13:58:19.113404 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.113317    2548 scope.go:117] "RemoveContainer" containerID="8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966"
Sep 16 13:58:19.122285 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.122265    2548 scope.go:117] "RemoveContainer" containerID="591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480"
Sep 16 13:58:19.130190 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.130171    2548 scope.go:117] "RemoveContainer" containerID="8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966"
Sep 16 13:58:19.130501 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:58:19.130483    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966\": container with ID starting with 8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966 not found: ID does not exist" containerID="8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966"
Sep 16 13:58:19.130573 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.130510    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966"} err="failed to get container status \"8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966\": rpc error: code = NotFound desc = could not find container \"8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966\": container with ID starting with 8d0cf30bbc59b12cda7ffbe4bc2d515128abf61a0249e88df1737427943e0966 not found: ID does not exist"
Sep 16 13:58:19.130573 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.130528    2548 scope.go:117] "RemoveContainer" containerID="591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480"
Sep 16 13:58:19.130743 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:58:19.130726    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480\": container with ID starting with 591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480 not found: ID does not exist" containerID="591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480"
Sep 16 13:58:19.130784 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.130747    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480"} err="failed to get container status \"591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480\": rpc error: code = NotFound desc = could not find container \"591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480\": container with ID starting with 591ef2680e1c1e00554c05575092f48657132e83afb724adcc7870a181963480 not found: ID does not exist"
Sep 16 13:58:19.137054 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.137030    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"]
Sep 16 13:58:19.142230 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.142212    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-38e8bf38-openshift-default-6c9f4bc7cd-7pmrf"]
Sep 16 13:58:19.970744 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.970708    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7" path="/var/lib/kubelet/pods/6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7/volumes"
Sep 16 13:58:19.971097 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.971046    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f025ac4-29db-408d-b8e3-7e797e4227b6" path="/var/lib/kubelet/pods/7f025ac4-29db-408d-b8e3-7e797e4227b6/volumes"
Sep 16 13:58:19.971441 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:19.971427    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" path="/var/lib/kubelet/pods/e924a7b6-0eb0-436f-a70b-e1433f0460ee/volumes"
Sep 16 13:58:23.071502 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.071469    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"]
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072419    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7" containerName="revoke-keys"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072459    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7" containerName="revoke-keys"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072490    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f025ac4-29db-408d-b8e3-7e797e4227b6" containerName="payload-processing"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072501    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f025ac4-29db-408d-b8e3-7e797e4227b6" containerName="payload-processing"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072549    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072559    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072781    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f025ac4-29db-408d-b8e3-7e797e4227b6" containerName="payload-processing"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072803    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072814    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy"
Sep 16 13:58:23.074647 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.072834    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2c4d24-b889-4b8a-88bd-2f82e7f42bd7" containerName="revoke-keys"
Sep 16 13:58:23.078673 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.078645    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:23.081980 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.081960    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1852eb66-dockercfg-44nl6\""
Sep 16 13:58:23.083493 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.083350    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1852eb66\""
Sep 16 13:58:23.086703 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.086680    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"]
Sep 16 13:58:23.128115 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.128090    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"]
Sep 16 13:58:23.128494 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.128481    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy"
Sep 16 13:58:23.128544 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.128496    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="e924a7b6-0eb0-436f-a70b-e1433f0460ee" containerName="istio-proxy"
Sep 16 13:58:23.131602 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.131584    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:23.143550 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.143527    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"]
Sep 16 13:58:23.215260 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.215221    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269fdb89-c66a-48d0-b228-557b0521910f-plugins-config-volume\") pod \"payload-processing-e2e-derive-1852eb66-76997f4669-2jflw\" (UID: \"269fdb89-c66a-48d0-b228-557b0521910f\") " pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:23.215415 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.215358    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91124115-5e31-4244-90d7-ad49dc9b4a7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz\" (UID: \"91124115-5e31-4244-90d7-ad49dc9b4a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:23.215464 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.215426    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb887\" (UniqueName: \"kubernetes.io/projected/91124115-5e31-4244-90d7-ad49dc9b4a7d-kube-api-access-cb887\") pod \"payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz\" (UID: \"91124115-5e31-4244-90d7-ad49dc9b4a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:23.215499 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.215458    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hc9t\" (UniqueName: \"kubernetes.io/projected/269fdb89-c66a-48d0-b228-557b0521910f-kube-api-access-9hc9t\") pod \"payload-processing-e2e-derive-1852eb66-76997f4669-2jflw\" (UID: \"269fdb89-c66a-48d0-b228-557b0521910f\") " pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:23.316731 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.316696    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb887\" (UniqueName: \"kubernetes.io/projected/91124115-5e31-4244-90d7-ad49dc9b4a7d-kube-api-access-cb887\") pod \"payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz\" (UID: \"91124115-5e31-4244-90d7-ad49dc9b4a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:23.316906 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.316745    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hc9t\" (UniqueName: \"kubernetes.io/projected/269fdb89-c66a-48d0-b228-557b0521910f-kube-api-access-9hc9t\") pod \"payload-processing-e2e-derive-1852eb66-76997f4669-2jflw\" (UID: \"269fdb89-c66a-48d0-b228-557b0521910f\") " pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:23.316906 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.316793    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269fdb89-c66a-48d0-b228-557b0521910f-plugins-config-volume\") pod \"payload-processing-e2e-derive-1852eb66-76997f4669-2jflw\" (UID: \"269fdb89-c66a-48d0-b228-557b0521910f\") " pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:23.316906 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.316861    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91124115-5e31-4244-90d7-ad49dc9b4a7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz\" (UID: \"91124115-5e31-4244-90d7-ad49dc9b4a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:23.317558 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.317533    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269fdb89-c66a-48d0-b228-557b0521910f-plugins-config-volume\") pod \"payload-processing-e2e-derive-1852eb66-76997f4669-2jflw\" (UID: \"269fdb89-c66a-48d0-b228-557b0521910f\") " pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:23.317656 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.317573    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91124115-5e31-4244-90d7-ad49dc9b4a7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz\" (UID: \"91124115-5e31-4244-90d7-ad49dc9b4a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:23.328519 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.328460    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb887\" (UniqueName: \"kubernetes.io/projected/91124115-5e31-4244-90d7-ad49dc9b4a7d-kube-api-access-cb887\") pod \"payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz\" (UID: \"91124115-5e31-4244-90d7-ad49dc9b4a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:23.328519 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.328460    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hc9t\" (UniqueName: \"kubernetes.io/projected/269fdb89-c66a-48d0-b228-557b0521910f-kube-api-access-9hc9t\") pod \"payload-processing-e2e-derive-1852eb66-76997f4669-2jflw\" (UID: \"269fdb89-c66a-48d0-b228-557b0521910f\") " pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:23.392427 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.392396    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:23.441202 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.440970    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:23.666603 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.666575    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"]
Sep 16 13:58:23.668241 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:58:23.668212    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269fdb89_c66a_48d0_b228_557b0521910f.slice/crio-e7bf7e71c2e9ccd9ccc63ef9ab8f9fe5900446b9c999f3deab1b8433c34c8550 WatchSource:0}: Error finding container e7bf7e71c2e9ccd9ccc63ef9ab8f9fe5900446b9c999f3deab1b8433c34c8550: Status 404 returned error can't find the container with id e7bf7e71c2e9ccd9ccc63ef9ab8f9fe5900446b9c999f3deab1b8433c34c8550
Sep 16 13:58:23.720317 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:23.720172    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"]
Sep 16 13:58:23.722781 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:58:23.722757    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91124115_5e31_4244_90d7_ad49dc9b4a7d.slice/crio-38fafe72d3d711b1b2d661bf7af7bd8a89b8bdd5163b7b0717393d0583e8ec26 WatchSource:0}: Error finding container 38fafe72d3d711b1b2d661bf7af7bd8a89b8bdd5163b7b0717393d0583e8ec26: Status 404 returned error can't find the container with id 38fafe72d3d711b1b2d661bf7af7bd8a89b8bdd5163b7b0717393d0583e8ec26
Sep 16 13:58:24.140360 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:24.140320    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw" event={"ID":"269fdb89-c66a-48d0-b228-557b0521910f","Type":"ContainerStarted","Data":"8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a"}
Sep 16 13:58:24.140806 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:24.140390    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:24.140806 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:24.140414    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw" event={"ID":"269fdb89-c66a-48d0-b228-557b0521910f","Type":"ContainerStarted","Data":"e7bf7e71c2e9ccd9ccc63ef9ab8f9fe5900446b9c999f3deab1b8433c34c8550"}
Sep 16 13:58:24.141975 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:24.141949    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz" event={"ID":"91124115-5e31-4244-90d7-ad49dc9b4a7d","Type":"ContainerStarted","Data":"b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a"}
Sep 16 13:58:24.142070 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:24.141998    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz" event={"ID":"91124115-5e31-4244-90d7-ad49dc9b4a7d","Type":"ContainerStarted","Data":"38fafe72d3d711b1b2d661bf7af7bd8a89b8bdd5163b7b0717393d0583e8ec26"}
Sep 16 13:58:24.142070 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:24.142064    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:24.165299 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:24.165233    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw" podStartSLOduration=1.165218929 podStartE2EDuration="1.165218929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:58:24.164204627 +0000 UTC m=+926.791686002" watchObservedRunningTime="2026-09-16 13:58:24.165218929 +0000 UTC m=+926.792700304"
Sep 16 13:58:24.188129 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:24.188088    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz" podStartSLOduration=1.188075361 podStartE2EDuration="1.188075361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:58:24.185365395 +0000 UTC m=+926.812846769" watchObservedRunningTime="2026-09-16 13:58:24.188075361 +0000 UTC m=+926.815556734"
Sep 16 13:58:35.148989 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:35.148956    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:58:35.149539 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:35.149269    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:58:59.798828 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:59.798793    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"]
Sep 16 13:58:59.799343 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:59.799037    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw" podUID="269fdb89-c66a-48d0-b228-557b0521910f" containerName="payload-processing" containerID="cri-o://8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a" gracePeriod=30
Sep 16 13:58:59.830720 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:59.830692    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"]
Sep 16 13:58:59.830918 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:58:59.830894    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz" podUID="91124115-5e31-4244-90d7-ad49dc9b4a7d" containerName="payload-pre-processing" containerID="cri-o://b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a" gracePeriod=30
Sep 16 13:59:00.093977 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.092736    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:59:00.129184 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.129158    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:59:00.152333 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.152303    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hc9t\" (UniqueName: \"kubernetes.io/projected/269fdb89-c66a-48d0-b228-557b0521910f-kube-api-access-9hc9t\") pod \"269fdb89-c66a-48d0-b228-557b0521910f\" (UID: \"269fdb89-c66a-48d0-b228-557b0521910f\") "
Sep 16 13:59:00.152480 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.152390    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb887\" (UniqueName: \"kubernetes.io/projected/91124115-5e31-4244-90d7-ad49dc9b4a7d-kube-api-access-cb887\") pod \"91124115-5e31-4244-90d7-ad49dc9b4a7d\" (UID: \"91124115-5e31-4244-90d7-ad49dc9b4a7d\") "
Sep 16 13:59:00.152480 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.152451    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91124115-5e31-4244-90d7-ad49dc9b4a7d-plugins-config-volume\") pod \"91124115-5e31-4244-90d7-ad49dc9b4a7d\" (UID: \"91124115-5e31-4244-90d7-ad49dc9b4a7d\") "
Sep 16 13:59:00.152606 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.152482    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269fdb89-c66a-48d0-b228-557b0521910f-plugins-config-volume\") pod \"269fdb89-c66a-48d0-b228-557b0521910f\" (UID: \"269fdb89-c66a-48d0-b228-557b0521910f\") "
Sep 16 13:59:00.152880 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.152848    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91124115-5e31-4244-90d7-ad49dc9b4a7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91124115-5e31-4244-90d7-ad49dc9b4a7d" (UID: "91124115-5e31-4244-90d7-ad49dc9b4a7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:59:00.152880 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.152860    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269fdb89-c66a-48d0-b228-557b0521910f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "269fdb89-c66a-48d0-b228-557b0521910f" (UID: "269fdb89-c66a-48d0-b228-557b0521910f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:59:00.154682 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.154659    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91124115-5e31-4244-90d7-ad49dc9b4a7d-kube-api-access-cb887" (OuterVolumeSpecName: "kube-api-access-cb887") pod "91124115-5e31-4244-90d7-ad49dc9b4a7d" (UID: "91124115-5e31-4244-90d7-ad49dc9b4a7d"). InnerVolumeSpecName "kube-api-access-cb887". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:00.154850 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.154827    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269fdb89-c66a-48d0-b228-557b0521910f-kube-api-access-9hc9t" (OuterVolumeSpecName: "kube-api-access-9hc9t") pod "269fdb89-c66a-48d0-b228-557b0521910f" (UID: "269fdb89-c66a-48d0-b228-557b0521910f"). InnerVolumeSpecName "kube-api-access-9hc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:00.254198 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.254165    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb887\" (UniqueName: \"kubernetes.io/projected/91124115-5e31-4244-90d7-ad49dc9b4a7d-kube-api-access-cb887\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:59:00.254198 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.254196    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91124115-5e31-4244-90d7-ad49dc9b4a7d-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:59:00.254417 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.254210    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269fdb89-c66a-48d0-b228-557b0521910f-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:59:00.254417 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.254222    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hc9t\" (UniqueName: \"kubernetes.io/projected/269fdb89-c66a-48d0-b228-557b0521910f-kube-api-access-9hc9t\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 13:59:00.338091 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.338010    2548 generic.go:356] "Generic (PLEG): container finished" podID="269fdb89-c66a-48d0-b228-557b0521910f" containerID="8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a" exitCode=0
Sep 16 13:59:00.338091 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.338049    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"
Sep 16 13:59:00.338338 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.338093    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw" event={"ID":"269fdb89-c66a-48d0-b228-557b0521910f","Type":"ContainerDied","Data":"8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a"}
Sep 16 13:59:00.338338 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.338143    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw" event={"ID":"269fdb89-c66a-48d0-b228-557b0521910f","Type":"ContainerDied","Data":"e7bf7e71c2e9ccd9ccc63ef9ab8f9fe5900446b9c999f3deab1b8433c34c8550"}
Sep 16 13:59:00.338338 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.338171    2548 scope.go:117] "RemoveContainer" containerID="8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a"
Sep 16 13:59:00.339479 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.339458    2548 generic.go:356] "Generic (PLEG): container finished" podID="91124115-5e31-4244-90d7-ad49dc9b4a7d" containerID="b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a" exitCode=0
Sep 16 13:59:00.339589 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.339496    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"
Sep 16 13:59:00.339589 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.339557    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz" event={"ID":"91124115-5e31-4244-90d7-ad49dc9b4a7d","Type":"ContainerDied","Data":"b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a"}
Sep 16 13:59:00.339700 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.339603    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz" event={"ID":"91124115-5e31-4244-90d7-ad49dc9b4a7d","Type":"ContainerDied","Data":"38fafe72d3d711b1b2d661bf7af7bd8a89b8bdd5163b7b0717393d0583e8ec26"}
Sep 16 13:59:00.348921 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.348906    2548 scope.go:117] "RemoveContainer" containerID="8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a"
Sep 16 13:59:00.349236 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:59:00.349218    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a\": container with ID starting with 8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a not found: ID does not exist" containerID="8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a"
Sep 16 13:59:00.349345 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.349260    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a"} err="failed to get container status \"8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a\": rpc error: code = NotFound desc = could not find container \"8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a\": container with ID starting with 8c7e7dc9f59c90896c5ec90bafeaea0b937086e8b565ab6c56c0241072e8460a not found: ID does not exist"
Sep 16 13:59:00.349345 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.349282    2548 scope.go:117] "RemoveContainer" containerID="b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a"
Sep 16 13:59:00.357838 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.357818    2548 scope.go:117] "RemoveContainer" containerID="b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a"
Sep 16 13:59:00.358091 ip-10-0-129-226 kubenswrapper[2548]: E0916 13:59:00.358073    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a\": container with ID starting with b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a not found: ID does not exist" containerID="b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a"
Sep 16 13:59:00.358151 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.358097    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a"} err="failed to get container status \"b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a\": rpc error: code = NotFound desc = could not find container \"b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a\": container with ID starting with b7c0a4e878df95a137d71d8c054a904dca45f92b6c607c071710d8cd46b1495a not found: ID does not exist"
Sep 16 13:59:00.386257 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.386217    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"]
Sep 16 13:59:00.394793 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.394769    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1852eb66-7cd99c8d6b-wqqxz"]
Sep 16 13:59:00.421063 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.421036    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"]
Sep 16 13:59:00.436143 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:00.436116    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1852eb66-76997f4669-2jflw"]
Sep 16 13:59:01.971101 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:01.971066    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="269fdb89-c66a-48d0-b228-557b0521910f" path="/var/lib/kubelet/pods/269fdb89-c66a-48d0-b228-557b0521910f/volumes"
Sep 16 13:59:01.971472 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:01.971418    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91124115-5e31-4244-90d7-ad49dc9b4a7d" path="/var/lib/kubelet/pods/91124115-5e31-4244-90d7-ad49dc9b4a7d/volumes"
Sep 16 13:59:04.900669 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.900639    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"]
Sep 16 13:59:04.901083 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.901068    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269fdb89-c66a-48d0-b228-557b0521910f" containerName="payload-processing"
Sep 16 13:59:04.901132 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.901085    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="269fdb89-c66a-48d0-b228-557b0521910f" containerName="payload-processing"
Sep 16 13:59:04.901132 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.901109    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91124115-5e31-4244-90d7-ad49dc9b4a7d" containerName="payload-pre-processing"
Sep 16 13:59:04.901132 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.901115    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="91124115-5e31-4244-90d7-ad49dc9b4a7d" containerName="payload-pre-processing"
Sep 16 13:59:04.901225 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.901171    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="269fdb89-c66a-48d0-b228-557b0521910f" containerName="payload-processing"
Sep 16 13:59:04.901225 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.901179    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="91124115-5e31-4244-90d7-ad49dc9b4a7d" containerName="payload-pre-processing"
Sep 16 13:59:04.906103 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.906075    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:04.908896 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.908874    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-733e5b-openshift-default-dockercfg-xv78p\""
Sep 16 13:59:04.922506 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:04.922483    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"]
Sep 16 13:59:05.003497 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.003456    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istiod-ca-cert\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.003812 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.003794    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-socket\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.003946 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.003931    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-credential-socket\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.004046 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.004034    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-token\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.004148 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.004136    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-certs\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.004298 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.004284    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-envoy\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.004449 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.004437    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w874m\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-kube-api-access-w874m\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.004551 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.004539    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-data\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.004666 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.004653    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-podinfo\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106277 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106217    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-data\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106468 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106306    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-podinfo\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106468 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106365    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istiod-ca-cert\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106468 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106409    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-socket\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106468 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106452    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-credential-socket\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106475    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-token\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106503    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-certs\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106607    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-envoy\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.106696 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.106672    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w874m\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-kube-api-access-w874m\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.107195 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.107168    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-socket\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.107474 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.107452    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-credential-socket\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.107582 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.107533    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-data\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.107795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.107766    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-certs\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.107795 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.107788    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istiod-ca-cert\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.109039 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.109018    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-envoy\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.109868 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.109849    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-podinfo\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.117970 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.117946    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-token\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.118371 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.118348    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w874m\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-kube-api-access-w874m\") pod \"e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") " pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.215997 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.215911    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:05.505830 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.505731    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"]
Sep 16 13:59:05.508643 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:59:05.508617    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacdf3c9_b5f5_40b8_97fb_b0a41e14b74f.slice/crio-0bfc7fa60a0ce32043102931ce74e573087f99d6c3576ca666a5bb73fb19c5e1 WatchSource:0}: Error finding container 0bfc7fa60a0ce32043102931ce74e573087f99d6c3576ca666a5bb73fb19c5e1: Status 404 returned error can't find the container with id 0bfc7fa60a0ce32043102931ce74e573087f99d6c3576ca666a5bb73fb19c5e1
Sep 16 13:59:05.510948 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.510910    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:59:05.511024 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.510986    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:59:05.511024 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:05.511017    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:59:06.373723 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:06.373684    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" event={"ID":"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f","Type":"ContainerStarted","Data":"d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774"}
Sep 16 13:59:06.374095 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:06.373737    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" event={"ID":"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f","Type":"ContainerStarted","Data":"0bfc7fa60a0ce32043102931ce74e573087f99d6c3576ca666a5bb73fb19c5e1"}
Sep 16 13:59:06.402206 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:06.402159    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podStartSLOduration=2.4021440370000002 podStartE2EDuration="2.402144037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:06.401372879 +0000 UTC m=+969.028854254" watchObservedRunningTime="2026-09-16 13:59:06.402144037 +0000 UTC m=+969.029625411"
Sep 16 13:59:07.216655 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:07.216617    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:07.221878 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:07.221851    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:07.379669 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:07.379637    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:07.380651 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:07.380633    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:28.492450 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:28.492415    2548 generic.go:356] "Generic (PLEG): container finished" podID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerID="d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774" exitCode=0
Sep 16 13:59:28.492873 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:28.492487    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" event={"ID":"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f","Type":"ContainerDied","Data":"d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774"}
Sep 16 13:59:28.492921 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:28.492881    2548 scope.go:117] "RemoveContainer" containerID="d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774"
Sep 16 13:59:28.493934 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:28.493904    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:59:28.494033 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:28.493969    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:59:28.494033 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:28.493998    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 13:59:29.500004 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:29.499970    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" event={"ID":"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f","Type":"ContainerStarted","Data":"b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb"}
Sep 16 13:59:30.216910 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:30.216872    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:30.217179 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:30.217111    2548 patch_prober.go:28] interesting pod/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 16 13:59:30.217226 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:30.217172    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 16 13:59:31.216302 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:31.216239    2548 patch_prober.go:28] interesting pod/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 16 13:59:31.216675 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:31.216327    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 16 13:59:32.217117 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:32.217077    2548 patch_prober.go:28] interesting pod/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 16 13:59:32.217516 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:32.217131    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 16 13:59:33.216965 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:33.216925    2548 patch_prober.go:28] interesting pod/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 16 13:59:33.217142 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:33.216985    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 16 13:59:34.216602 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:34.216564    2548 patch_prober.go:28] interesting pod/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 16 13:59:34.216797 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:34.216626    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 16 13:59:35.216331 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:35.216293    2548 patch_prober.go:28] interesting pod/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 16 13:59:35.216742 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:35.216364    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 16 13:59:36.220849 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:36.220766    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:36.221304 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:36.221136    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:36.221930 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:36.221910    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 13:59:40.554400 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.554363    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"]
Sep 16 13:59:40.558452 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.558431    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.562481 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.562456    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-95743e\"/\"openshift-service-ca.crt\""
Sep 16 13:59:40.562612 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.562467    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-95743e\"/\"test-model-95743e-kserve-self-signed-certs\""
Sep 16 13:59:40.562612 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.562506    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-95743e\"/\"default-dockercfg-ndlk4\""
Sep 16 13:59:40.562612 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.562509    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-95743e\"/\"kube-root-ca.crt\""
Sep 16 13:59:40.568604 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.568582    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"]
Sep 16 13:59:40.665173 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.665140    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-home\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.665173 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.665175    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tls-certs\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.665401 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.665206    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-dshm\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.665401 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.665275    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4px2\" (UniqueName: \"kubernetes.io/projected/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-kube-api-access-p4px2\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.665401 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.665357    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tmp-dir\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.665515 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.665403    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-model-cache\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766362 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766335    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tmp-dir\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766521 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766392    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-model-cache\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766521 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766436    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-home\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766521 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766464    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tls-certs\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766521 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766505    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-dshm\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766749 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766530    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4px2\" (UniqueName: \"kubernetes.io/projected/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-kube-api-access-p4px2\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766807 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766786    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tmp-dir\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766860 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766840    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-model-cache\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.766919 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.766877    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-home\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.768850 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.768827    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-dshm\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.769356 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.769335    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tls-certs\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.775736 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.775715    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4px2\" (UniqueName: \"kubernetes.io/projected/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-kube-api-access-p4px2\") pod \"test-model-95743e-kserve-7cf8988bc5-j2bqf\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") " pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:40.869134 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:40.869105    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:41.147497 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:41.147474    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"]
Sep 16 13:59:41.149035 ip-10-0-129-226 kubenswrapper[2548]: W0916 13:59:41.149005    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8eb68f45_7a1f_45b9_a71c_5465ee4153bd.slice/crio-b581b1e4c656b7ef5c7b9122f86ee90fbaaf30cb9d683f9a4bc182104cdb8fb6 WatchSource:0}: Error finding container b581b1e4c656b7ef5c7b9122f86ee90fbaaf30cb9d683f9a4bc182104cdb8fb6: Status 404 returned error can't find the container with id b581b1e4c656b7ef5c7b9122f86ee90fbaaf30cb9d683f9a4bc182104cdb8fb6
Sep 16 13:59:41.576044 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:41.575948    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf" event={"ID":"8eb68f45-7a1f-45b9-a71c-5465ee4153bd","Type":"ContainerStarted","Data":"c226a902bde28c1d9be0d81b292032eab312650046c4976fff6f48afb631ad78"}
Sep 16 13:59:41.576044 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:41.575997    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf" event={"ID":"8eb68f45-7a1f-45b9-a71c-5465ee4153bd","Type":"ContainerStarted","Data":"b581b1e4c656b7ef5c7b9122f86ee90fbaaf30cb9d683f9a4bc182104cdb8fb6"}
Sep 16 13:59:41.602094 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:41.602049    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf" podStartSLOduration=1.602033155 podStartE2EDuration="1.602033155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:41.599902758 +0000 UTC m=+1004.227384146" watchObservedRunningTime="2026-09-16 13:59:41.602033155 +0000 UTC m=+1004.229514528"
Sep 16 13:59:50.869564 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:50.869523    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:50.869564 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:50.869574    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:50.882659 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:50.882635    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:51.639203 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:51.639175    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 13:59:59.541650 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.541614    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"]
Sep 16 13:59:59.546371 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.546348    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.548945 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.548922    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-733e5b\"/\"default-dockercfg-9prjv\""
Sep 16 13:59:59.549046 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.548978    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-733e5b\"/\"explicit-model-733e5b-kserve-self-signed-certs\""
Sep 16 13:59:59.549046 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.549007    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-733e5b\"/\"openshift-service-ca.crt\""
Sep 16 13:59:59.549046 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.548978    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-733e5b\"/\"kube-root-ca.crt\""
Sep 16 13:59:59.554905 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.554885    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"]
Sep 16 13:59:59.656471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.656433    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-home\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.656471 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.656474    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-dshm\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.656668 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.656494    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-tmp-dir\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.656668 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.656612    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5sm\" (UniqueName: \"kubernetes.io/projected/67796e79-195d-4d58-81dc-7d91d6c1c51a-kube-api-access-rg5sm\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.656735 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.656676    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-model-cache\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.656735 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.656694    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67796e79-195d-4d58-81dc-7d91d6c1c51a-tls-certs\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.757420 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757390    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-home\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.757585 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757438    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-dshm\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.757585 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757460    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-tmp-dir\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.757585 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757514    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5sm\" (UniqueName: \"kubernetes.io/projected/67796e79-195d-4d58-81dc-7d91d6c1c51a-kube-api-access-rg5sm\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.757585 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757561    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-model-cache\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.757767 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757585    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67796e79-195d-4d58-81dc-7d91d6c1c51a-tls-certs\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.757924 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757894    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-tmp-dir\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.757924 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757911    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-home\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.758078 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.757952    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-model-cache\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.759848 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.759820    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-dshm\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.760428 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.760392    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67796e79-195d-4d58-81dc-7d91d6c1c51a-tls-certs\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.769187 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.769162    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5sm\" (UniqueName: \"kubernetes.io/projected/67796e79-195d-4d58-81dc-7d91d6c1c51a-kube-api-access-rg5sm\") pod \"explicit-model-733e5b-kserve-75957f94c5-8nzqz\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") " pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 13:59:59.856414 ip-10-0-129-226 kubenswrapper[2548]: I0916 13:59:59.856065    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 14:00:00.141705 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.141675    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"]
Sep 16 14:00:00.143851 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:00:00.143814    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67796e79_195d_4d58_81dc_7d91d6c1c51a.slice/crio-6a81b4c6a37c92d86f188268bbf8219b45275f55976de9c8f73eea15d2efeec1 WatchSource:0}: Error finding container 6a81b4c6a37c92d86f188268bbf8219b45275f55976de9c8f73eea15d2efeec1: Status 404 returned error can't find the container with id 6a81b4c6a37c92d86f188268bbf8219b45275f55976de9c8f73eea15d2efeec1
Sep 16 14:00:00.153384 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.153362    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n"]
Sep 16 14:00:00.156803 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.156788    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n"
Sep 16 14:00:00.157077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.157054    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"]
Sep 16 14:00:00.160160 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.160144    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"
Sep 16 14:00:00.160877 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.160857    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:00:00.160961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.160926    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"]
Sep 16 14:00:00.181066 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.173181    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"
Sep 16 14:00:00.181066 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.175142    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n"]
Sep 16 14:00:00.181066 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.178214    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"]
Sep 16 14:00:00.181529 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.181418    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n"
Sep 16 14:00:00.194120 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.194094    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"
Sep 16 14:00:00.196917 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.196897    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"]
Sep 16 14:00:00.211827 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.211798    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"
Sep 16 14:00:00.484327 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.484293    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"]
Sep 16 14:00:00.486137 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:00:00.486107    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffdf8fc_0843_494f_8b41_7b84e5e7d6ec.slice/crio-54834010c543fa7c33363161fe0bd7eb59bcba840b41d5cd7509ba81859b5fd5 WatchSource:0}: Error finding container 54834010c543fa7c33363161fe0bd7eb59bcba840b41d5cd7509ba81859b5fd5: Status 404 returned error can't find the container with id 54834010c543fa7c33363161fe0bd7eb59bcba840b41d5cd7509ba81859b5fd5
Sep 16 14:00:00.486413 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.486393    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n"]
Sep 16 14:00:00.487161 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:00:00.487140    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d852a22_3f3f_4b75_96b4_e44f27488976.slice/crio-9356bfed958af1430feedf46f5a131e916a57d9cdeaf8f6559caa163d973c661 WatchSource:0}: Error finding container 9356bfed958af1430feedf46f5a131e916a57d9cdeaf8f6559caa163d973c661: Status 404 returned error can't find the container with id 9356bfed958af1430feedf46f5a131e916a57d9cdeaf8f6559caa163d973c661
Sep 16 14:00:00.527403 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.527318    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"]
Sep 16 14:00:00.529513 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:00:00.529489    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50efba3a_ebba_447b_8a86_f629209a63b7.slice/crio-0d79626caf06fdc4b4f6b000feed68888f37b1dd9576458938f28c5a0ce68ac4 WatchSource:0}: Error finding container 0d79626caf06fdc4b4f6b000feed68888f37b1dd9576458938f28c5a0ce68ac4: Status 404 returned error can't find the container with id 0d79626caf06fdc4b4f6b000feed68888f37b1dd9576458938f28c5a0ce68ac4
Sep 16 14:00:00.675022 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.674988    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c" event={"ID":"2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec","Type":"ContainerStarted","Data":"54834010c543fa7c33363161fe0bd7eb59bcba840b41d5cd7509ba81859b5fd5"}
Sep 16 14:00:00.676202 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.676176    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v" event={"ID":"50efba3a-ebba-447b-8a86-f629209a63b7","Type":"ContainerStarted","Data":"0d79626caf06fdc4b4f6b000feed68888f37b1dd9576458938f28c5a0ce68ac4"}
Sep 16 14:00:00.678306 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.678277    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz" event={"ID":"67796e79-195d-4d58-81dc-7d91d6c1c51a","Type":"ContainerStarted","Data":"ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1"}
Sep 16 14:00:00.678426 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.678319    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz" event={"ID":"67796e79-195d-4d58-81dc-7d91d6c1c51a","Type":"ContainerStarted","Data":"6a81b4c6a37c92d86f188268bbf8219b45275f55976de9c8f73eea15d2efeec1"}
Sep 16 14:00:00.679744 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.679710    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n" event={"ID":"8d852a22-3f3f-4b75-96b4-e44f27488976","Type":"ContainerStarted","Data":"9356bfed958af1430feedf46f5a131e916a57d9cdeaf8f6559caa163d973c661"}
Sep 16 14:00:00.716819 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:00.716779    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz" podStartSLOduration=1.716764123 podStartE2EDuration="1.716764123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:00.715931979 +0000 UTC m=+1023.343413365" watchObservedRunningTime="2026-09-16 14:00:00.716764123 +0000 UTC m=+1023.344245496"
Sep 16 14:00:01.687311 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:01.687279    2548 generic.go:356] "Generic (PLEG): container finished" podID="50efba3a-ebba-447b-8a86-f629209a63b7" containerID="95ae036a2b46dd6eaf42654b0fe61c39fbc1ebb76658d4413f015d01cb69d4d1" exitCode=0
Sep 16 14:00:01.687679 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:01.687367    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v" event={"ID":"50efba3a-ebba-447b-8a86-f629209a63b7","Type":"ContainerDied","Data":"95ae036a2b46dd6eaf42654b0fe61c39fbc1ebb76658d4413f015d01cb69d4d1"}
Sep 16 14:00:01.689189 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:01.689159    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n" event={"ID":"8d852a22-3f3f-4b75-96b4-e44f27488976","Type":"ContainerStarted","Data":"a298b9a46c2a5a03c00f4aaa2919e94393b1d5eb1234ef0a169694a17fe6906c"}
Sep 16 14:00:01.691007 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:01.690985    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c" event={"ID":"2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec","Type":"ContainerStarted","Data":"ef1c4642b361d70b380a6dcd2fbf770affa7abf9f18ad9a54138ff4a9e26a641"}
Sep 16 14:00:01.724285 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:01.724225    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n" podStartSLOduration=1.724210644 podStartE2EDuration="1.724210644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.721487268 +0000 UTC m=+1024.348968655" watchObservedRunningTime="2026-09-16 14:00:01.724210644 +0000 UTC m=+1024.351692036"
Sep 16 14:00:01.739128 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:01.739059    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c" podStartSLOduration=1.739045325 podStartE2EDuration="1.739045325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:01.738011959 +0000 UTC m=+1024.365493346" watchObservedRunningTime="2026-09-16 14:00:01.739045325 +0000 UTC m=+1024.366526699"
Sep 16 14:00:02.698348 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:02.698313    2548 generic.go:356] "Generic (PLEG): container finished" podID="8d852a22-3f3f-4b75-96b4-e44f27488976" containerID="a298b9a46c2a5a03c00f4aaa2919e94393b1d5eb1234ef0a169694a17fe6906c" exitCode=0
Sep 16 14:00:02.698742 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:02.698393    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n" event={"ID":"8d852a22-3f3f-4b75-96b4-e44f27488976","Type":"ContainerDied","Data":"a298b9a46c2a5a03c00f4aaa2919e94393b1d5eb1234ef0a169694a17fe6906c"}
Sep 16 14:00:02.699968 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:02.699946    2548 generic.go:356] "Generic (PLEG): container finished" podID="2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec" containerID="ef1c4642b361d70b380a6dcd2fbf770affa7abf9f18ad9a54138ff4a9e26a641" exitCode=0
Sep 16 14:00:02.700066 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:02.699971    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c" event={"ID":"2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec","Type":"ContainerDied","Data":"ef1c4642b361d70b380a6dcd2fbf770affa7abf9f18ad9a54138ff4a9e26a641"}
Sep 16 14:00:02.836147 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:02.836122    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"
Sep 16 14:00:03.707620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:03.707586    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"
Sep 16 14:00:03.708092 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:03.707631    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v" event={"ID":"50efba3a-ebba-447b-8a86-f629209a63b7","Type":"ContainerDied","Data":"0d79626caf06fdc4b4f6b000feed68888f37b1dd9576458938f28c5a0ce68ac4"}
Sep 16 14:00:03.708092 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:03.707720    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d79626caf06fdc4b4f6b000feed68888f37b1dd9576458938f28c5a0ce68ac4"
Sep 16 14:00:03.847671 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:03.847646    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"
Sep 16 14:00:03.896204 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:03.896176    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n"
Sep 16 14:00:04.714485 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:04.714453    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n"
Sep 16 14:00:04.714936 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:04.714498    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-4gz7n" event={"ID":"8d852a22-3f3f-4b75-96b4-e44f27488976","Type":"ContainerDied","Data":"9356bfed958af1430feedf46f5a131e916a57d9cdeaf8f6559caa163d973c661"}
Sep 16 14:00:04.714936 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:04.714546    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9356bfed958af1430feedf46f5a131e916a57d9cdeaf8f6559caa163d973c661"
Sep 16 14:00:04.715847 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:04.715828    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"
Sep 16 14:00:04.715847 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:04.715832    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c" event={"ID":"2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec","Type":"ContainerDied","Data":"54834010c543fa7c33363161fe0bd7eb59bcba840b41d5cd7509ba81859b5fd5"}
Sep 16 14:00:04.716003 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:04.715872    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54834010c543fa7c33363161fe0bd7eb59bcba840b41d5cd7509ba81859b5fd5"
Sep 16 14:00:09.856720 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:09.856685    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 14:00:09.857094 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:09.856735    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 14:00:09.869382 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:09.869357    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 14:00:10.760861 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:10.760833    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 14:00:15.153004 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:15.152969    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"]
Sep 16 14:00:30.171523 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.171482    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz" podUID="67796e79-195d-4d58-81dc-7d91d6c1c51a" containerName="main" containerID="cri-o://ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1" gracePeriod=45
Sep 16 14:00:30.413622 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.413599    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 14:00:30.567960 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.567876    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67796e79-195d-4d58-81dc-7d91d6c1c51a-tls-certs\") pod \"67796e79-195d-4d58-81dc-7d91d6c1c51a\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") "
Sep 16 14:00:30.567960 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.567947    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg5sm\" (UniqueName: \"kubernetes.io/projected/67796e79-195d-4d58-81dc-7d91d6c1c51a-kube-api-access-rg5sm\") pod \"67796e79-195d-4d58-81dc-7d91d6c1c51a\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") "
Sep 16 14:00:30.568174 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568006    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-tmp-dir\") pod \"67796e79-195d-4d58-81dc-7d91d6c1c51a\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") "
Sep 16 14:00:30.568174 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568067    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-model-cache\") pod \"67796e79-195d-4d58-81dc-7d91d6c1c51a\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") "
Sep 16 14:00:30.568174 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568090    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-home\") pod \"67796e79-195d-4d58-81dc-7d91d6c1c51a\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") "
Sep 16 14:00:30.568174 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568115    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-dshm\") pod \"67796e79-195d-4d58-81dc-7d91d6c1c51a\" (UID: \"67796e79-195d-4d58-81dc-7d91d6c1c51a\") "
Sep 16 14:00:30.568416 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568346    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-model-cache" (OuterVolumeSpecName: "model-cache") pod "67796e79-195d-4d58-81dc-7d91d6c1c51a" (UID: "67796e79-195d-4d58-81dc-7d91d6c1c51a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:30.568416 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568353    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67796e79-195d-4d58-81dc-7d91d6c1c51a" (UID: "67796e79-195d-4d58-81dc-7d91d6c1c51a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:30.568416 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568399    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-home" (OuterVolumeSpecName: "home") pod "67796e79-195d-4d58-81dc-7d91d6c1c51a" (UID: "67796e79-195d-4d58-81dc-7d91d6c1c51a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:30.568548 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568521    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:30.568589 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568555    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:30.568589 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.568567    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:30.570160 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.570138    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-dshm" (OuterVolumeSpecName: "dshm") pod "67796e79-195d-4d58-81dc-7d91d6c1c51a" (UID: "67796e79-195d-4d58-81dc-7d91d6c1c51a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:30.570291 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.570164    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67796e79-195d-4d58-81dc-7d91d6c1c51a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67796e79-195d-4d58-81dc-7d91d6c1c51a" (UID: "67796e79-195d-4d58-81dc-7d91d6c1c51a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:00:30.570291 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.570177    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67796e79-195d-4d58-81dc-7d91d6c1c51a-kube-api-access-rg5sm" (OuterVolumeSpecName: "kube-api-access-rg5sm") pod "67796e79-195d-4d58-81dc-7d91d6c1c51a" (UID: "67796e79-195d-4d58-81dc-7d91d6c1c51a"). InnerVolumeSpecName "kube-api-access-rg5sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:00:30.669649 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.669612    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67796e79-195d-4d58-81dc-7d91d6c1c51a-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:30.669649 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.669643    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg5sm\" (UniqueName: \"kubernetes.io/projected/67796e79-195d-4d58-81dc-7d91d6c1c51a-kube-api-access-rg5sm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:30.669649 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.669655    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67796e79-195d-4d58-81dc-7d91d6c1c51a-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:30.859990 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.859907    2548 generic.go:356] "Generic (PLEG): container finished" podID="67796e79-195d-4d58-81dc-7d91d6c1c51a" containerID="ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1" exitCode=0
Sep 16 14:00:30.859990 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.859956    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"
Sep 16 14:00:30.860169 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.859996    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz" event={"ID":"67796e79-195d-4d58-81dc-7d91d6c1c51a","Type":"ContainerDied","Data":"ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1"}
Sep 16 14:00:30.860169 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.860045    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz" event={"ID":"67796e79-195d-4d58-81dc-7d91d6c1c51a","Type":"ContainerDied","Data":"6a81b4c6a37c92d86f188268bbf8219b45275f55976de9c8f73eea15d2efeec1"}
Sep 16 14:00:30.860169 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.860063    2548 scope.go:117] "RemoveContainer" containerID="ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1"
Sep 16 14:00:30.869892 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.869865    2548 scope.go:117] "RemoveContainer" containerID="ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1"
Sep 16 14:00:30.870168 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:00:30.870147    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1\": container with ID starting with ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1 not found: ID does not exist" containerID="ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1"
Sep 16 14:00:30.870239 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.870179    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1"} err="failed to get container status \"ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1\": rpc error: code = NotFound desc = could not find container \"ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1\": container with ID starting with ecbc0cda641b7923cd33ab66b233a48227a8d650224fb1973f6a6b4adeeca3a1 not found: ID does not exist"
Sep 16 14:00:30.897764 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.897737    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"]
Sep 16 14:00:30.909367 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:30.909348    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-733e5b/explicit-model-733e5b-kserve-75957f94c5-8nzqz"]
Sep 16 14:00:31.970239 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:31.970205    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67796e79-195d-4d58-81dc-7d91d6c1c51a" path="/var/lib/kubelet/pods/67796e79-195d-4d58-81dc-7d91d6c1c51a/volumes"
Sep 16 14:00:33.922921 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:33.922883    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"]
Sep 16 14:00:33.923506 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:33.923098    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy" containerID="cri-o://b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb" gracePeriod=30
Sep 16 14:00:34.773316 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773284    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"]
Sep 16 14:00:34.773888 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773873    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d852a22-3f3f-4b75-96b4-e44f27488976" containerName="cleanup"
Sep 16 14:00:34.773933 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773891    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d852a22-3f3f-4b75-96b4-e44f27488976" containerName="cleanup"
Sep 16 14:00:34.773933 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773910    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50efba3a-ebba-447b-8a86-f629209a63b7" containerName="cleanup"
Sep 16 14:00:34.773933 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773919    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="50efba3a-ebba-447b-8a86-f629209a63b7" containerName="cleanup"
Sep 16 14:00:34.774030 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773934    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec" containerName="cleanup"
Sep 16 14:00:34.774030 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773943    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec" containerName="cleanup"
Sep 16 14:00:34.774030 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773956    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67796e79-195d-4d58-81dc-7d91d6c1c51a" containerName="main"
Sep 16 14:00:34.774030 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.773964    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="67796e79-195d-4d58-81dc-7d91d6c1c51a" containerName="main"
Sep 16 14:00:34.774156 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.774046    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec" containerName="cleanup"
Sep 16 14:00:34.774156 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.774063    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d852a22-3f3f-4b75-96b4-e44f27488976" containerName="cleanup"
Sep 16 14:00:34.774156 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.774073    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="67796e79-195d-4d58-81dc-7d91d6c1c51a" containerName="main"
Sep 16 14:00:34.774156 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.774081    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="50efba3a-ebba-447b-8a86-f629209a63b7" containerName="cleanup"
Sep 16 14:00:34.779079 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.779047    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.801624 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.801599    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-b45798-openshift-default-dockercfg-kk4x4\""
Sep 16 14:00:34.885240 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.885206    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"]
Sep 16 14:00:34.914459 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914424    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-socket\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.914607 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914471    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-envoy\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.914650 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914611    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-istio-token\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.914704 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914665    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621858aa-3e62-41e2-8da9-c6659cee462f-istio-podinfo\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.914760 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914737    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6swrt\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-kube-api-access-6swrt\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.914814 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914791    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-certs\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.914869 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914825    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621858aa-3e62-41e2-8da9-c6659cee462f-istiod-ca-cert\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.914869 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914851    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-credential-socket\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:34.914958 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:34.914889    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-data\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016372 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016323    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-certs\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016372 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016375    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621858aa-3e62-41e2-8da9-c6659cee462f-istiod-ca-cert\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016400    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-credential-socket\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016441    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-data\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016487    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-socket\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016522    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-envoy\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016607    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-istio-token\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016647    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621858aa-3e62-41e2-8da9-c6659cee462f-istio-podinfo\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016697    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6swrt\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-kube-api-access-6swrt\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.016887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016790    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-certs\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.017215 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016886    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-credential-socket\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.017215 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016908    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-data\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.017215 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.016957    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-socket\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.017374 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.017356    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621858aa-3e62-41e2-8da9-c6659cee462f-istiod-ca-cert\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.018931 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.018911    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-envoy\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.019136 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.019119    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621858aa-3e62-41e2-8da9-c6659cee462f-istio-podinfo\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.036161 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.036111    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-istio-token\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.052776 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.052758    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6swrt\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-kube-api-access-6swrt\") pod \"e2e-noten-b45798-openshift-default-7f54f884b-4hq9s\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") " pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.089663 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.089639    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:35.375315 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.375288    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"]
Sep 16 14:00:35.379821 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:00:35.379784    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621858aa_3e62_41e2_8da9_c6659cee462f.slice/crio-00649a74a4b4e9d7badeff5ad877e6b94a82b9bea0bbc76cf1b0cbd605687149 WatchSource:0}: Error finding container 00649a74a4b4e9d7badeff5ad877e6b94a82b9bea0bbc76cf1b0cbd605687149: Status 404 returned error can't find the container with id 00649a74a4b4e9d7badeff5ad877e6b94a82b9bea0bbc76cf1b0cbd605687149
Sep 16 14:00:35.381988 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.381953    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:00:35.382094 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.382020    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:00:35.382094 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.382052    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:00:35.890705 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.890668    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s" event={"ID":"621858aa-3e62-41e2-8da9-c6659cee462f","Type":"ContainerStarted","Data":"1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe"}
Sep 16 14:00:35.890892 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.890727    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s" event={"ID":"621858aa-3e62-41e2-8da9-c6659cee462f","Type":"ContainerStarted","Data":"00649a74a4b4e9d7badeff5ad877e6b94a82b9bea0bbc76cf1b0cbd605687149"}
Sep 16 14:00:35.916561 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:35.916513    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s" podStartSLOduration=1.916496725 podStartE2EDuration="1.916496725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:35.915035067 +0000 UTC m=+1058.542516442" watchObservedRunningTime="2026-09-16 14:00:35.916496725 +0000 UTC m=+1058.543978098"
Sep 16 14:00:36.090492 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:36.090458    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:36.095181 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:36.095158    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:36.223549 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:36.223476    2548 patch_prober.go:28] interesting pod/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:00:36.223549 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:36.223525    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:00:36.896688 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:36.896660    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:36.897517 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:36.897493    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:00:39.179431 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.179407    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 14:00:39.263014 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.262925    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-credential-socket\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.263014 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.262981    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-token\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.263014 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263008    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-certs\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.263344 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263026    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-data\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.263344 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263058    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w874m\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-kube-api-access-w874m\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.263344 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263088    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-socket\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.263344 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263128    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-podinfo\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.263344 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263211    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-envoy\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.263344 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263321    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:39.263692 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263546    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-data" (OuterVolumeSpecName: "istio-data") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:39.263692 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263622    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.263692 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263644    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.263870 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263734    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:39.263870 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.263762    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:39.266237 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.266196    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:39.266237 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.266207    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:00:39.266237 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.266207    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-kube-api-access-w874m" (OuterVolumeSpecName: "kube-api-access-w874m") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "kube-api-access-w874m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:00:39.266533 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.266294    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-token" (OuterVolumeSpecName: "istio-token") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:00:39.364020 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.363995    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istiod-ca-cert\") pod \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\" (UID: \"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f\") "
Sep 16 14:00:39.364191 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.364180    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.364235 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.364193    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.364235 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.364202    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w874m\" (UniqueName: \"kubernetes.io/projected/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-kube-api-access-w874m\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.364235 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.364211    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.364235 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.364219    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.364235 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.364228    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.364421 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.364309    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" (UID: "bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:00:39.465143 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.465120    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:00:39.921883 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.921845    2548 generic.go:356] "Generic (PLEG): container finished" podID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerID="b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb" exitCode=0
Sep 16 14:00:39.922050 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.921898    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"
Sep 16 14:00:39.922050 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.921926    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" event={"ID":"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f","Type":"ContainerDied","Data":"b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb"}
Sep 16 14:00:39.922050 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.921999    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7" event={"ID":"bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f","Type":"ContainerDied","Data":"0bfc7fa60a0ce32043102931ce74e573087f99d6c3576ca666a5bb73fb19c5e1"}
Sep 16 14:00:39.922050 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.922024    2548 scope.go:117] "RemoveContainer" containerID="b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb"
Sep 16 14:00:39.931709 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.931690    2548 scope.go:117] "RemoveContainer" containerID="d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774"
Sep 16 14:00:39.940232 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.940217    2548 scope.go:117] "RemoveContainer" containerID="b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb"
Sep 16 14:00:39.940506 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:00:39.940490    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb\": container with ID starting with b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb not found: ID does not exist" containerID="b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb"
Sep 16 14:00:39.940550 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.940513    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb"} err="failed to get container status \"b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb\": rpc error: code = NotFound desc = could not find container \"b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb\": container with ID starting with b28b9044ea38483bc49ea16ddc79e8d2bc7e96693801d4ee6faa24b85deae7fb not found: ID does not exist"
Sep 16 14:00:39.940550 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.940529    2548 scope.go:117] "RemoveContainer" containerID="d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774"
Sep 16 14:00:39.940774 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:00:39.940761    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774\": container with ID starting with d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774 not found: ID does not exist" containerID="d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774"
Sep 16 14:00:39.940809 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.940777    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774"} err="failed to get container status \"d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774\": rpc error: code = NotFound desc = could not find container \"d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774\": container with ID starting with d77d4b0ea66820d5eb724aad491ffd9f3fa9d40a8f87964109bdd8e3e7bc3774 not found: ID does not exist"
Sep 16 14:00:39.946671 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.946653    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"]
Sep 16 14:00:39.953501 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.953479    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-733e5b-openshift-default-69776d5d56-nbfq7"]
Sep 16 14:00:39.969677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:39.969654    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" path="/var/lib/kubelet/pods/bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f/volumes"
Sep 16 14:00:49.500809 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.500773    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"]
Sep 16 14:00:49.501190 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.501176    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy"
Sep 16 14:00:49.501190 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.501191    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy"
Sep 16 14:00:49.501309 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.501208    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy"
Sep 16 14:00:49.501309 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.501214    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy"
Sep 16 14:00:49.501309 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.501299    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy"
Sep 16 14:00:49.501309 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.501310    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="bacdf3c9-b5f5-40b8-97fb-b0a41e14b74f" containerName="istio-proxy"
Sep 16 14:00:49.505626 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.505609    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:49.508086 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.508057    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2b5aadeb\""
Sep 16 14:00:49.515284 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.515241    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"]
Sep 16 14:00:49.519123 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.519102    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"]
Sep 16 14:00:49.519234 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.519191    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:49.521872 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.521850    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2b5aadeb-dockercfg-pschz\""
Sep 16 14:00:49.529617 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.529594    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"]
Sep 16 14:00:49.558171 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.558139    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd31bc5-8505-44f0-922a-6c36f513db0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj\" (UID: \"fdd31bc5-8505-44f0-922a-6c36f513db0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:49.558294 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.558183    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988lw\" (UniqueName: \"kubernetes.io/projected/fdd31bc5-8505-44f0-922a-6c36f513db0b-kube-api-access-988lw\") pod \"payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj\" (UID: \"fdd31bc5-8505-44f0-922a-6c36f513db0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:49.558342 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.558320    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98067113-7648-45e6-ba4d-52a598f36f01-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9\" (UID: \"98067113-7648-45e6-ba4d-52a598f36f01\") " pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:49.558384 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.558361    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx775\" (UniqueName: \"kubernetes.io/projected/98067113-7648-45e6-ba4d-52a598f36f01-kube-api-access-wx775\") pod \"payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9\" (UID: \"98067113-7648-45e6-ba4d-52a598f36f01\") " pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:49.659038 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.659009    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx775\" (UniqueName: \"kubernetes.io/projected/98067113-7648-45e6-ba4d-52a598f36f01-kube-api-access-wx775\") pod \"payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9\" (UID: \"98067113-7648-45e6-ba4d-52a598f36f01\") " pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:49.659181 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.659070    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd31bc5-8505-44f0-922a-6c36f513db0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj\" (UID: \"fdd31bc5-8505-44f0-922a-6c36f513db0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:49.659181 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.659090    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988lw\" (UniqueName: \"kubernetes.io/projected/fdd31bc5-8505-44f0-922a-6c36f513db0b-kube-api-access-988lw\") pod \"payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj\" (UID: \"fdd31bc5-8505-44f0-922a-6c36f513db0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:49.659181 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.659130    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98067113-7648-45e6-ba4d-52a598f36f01-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9\" (UID: \"98067113-7648-45e6-ba4d-52a598f36f01\") " pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:49.659891 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.659870    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98067113-7648-45e6-ba4d-52a598f36f01-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9\" (UID: \"98067113-7648-45e6-ba4d-52a598f36f01\") " pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:49.659891 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.659878    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd31bc5-8505-44f0-922a-6c36f513db0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj\" (UID: \"fdd31bc5-8505-44f0-922a-6c36f513db0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:49.670233 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.670203    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx775\" (UniqueName: \"kubernetes.io/projected/98067113-7648-45e6-ba4d-52a598f36f01-kube-api-access-wx775\") pod \"payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9\" (UID: \"98067113-7648-45e6-ba4d-52a598f36f01\") " pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:49.670348 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.670323    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-988lw\" (UniqueName: \"kubernetes.io/projected/fdd31bc5-8505-44f0-922a-6c36f513db0b-kube-api-access-988lw\") pod \"payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj\" (UID: \"fdd31bc5-8505-44f0-922a-6c36f513db0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:49.818325 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.818213    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:49.832398 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:49.832373    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:50.096686 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:50.096660    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"]
Sep 16 14:00:50.097907 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:00:50.097881    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd31bc5_8505_44f0_922a_6c36f513db0b.slice/crio-5e22a830d7a0fe945d081652033f35c9cfe30c570826d2fbc26cbafd42015927 WatchSource:0}: Error finding container 5e22a830d7a0fe945d081652033f35c9cfe30c570826d2fbc26cbafd42015927: Status 404 returned error can't find the container with id 5e22a830d7a0fe945d081652033f35c9cfe30c570826d2fbc26cbafd42015927
Sep 16 14:00:50.127511 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:50.127485    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"]
Sep 16 14:00:50.131362 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:00:50.131336    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98067113_7648_45e6_ba4d_52a598f36f01.slice/crio-74b7de6156775df4dd35b2b2c774172099827d3732f4e4be8b9528e0bf9ef796 WatchSource:0}: Error finding container 74b7de6156775df4dd35b2b2c774172099827d3732f4e4be8b9528e0bf9ef796: Status 404 returned error can't find the container with id 74b7de6156775df4dd35b2b2c774172099827d3732f4e4be8b9528e0bf9ef796
Sep 16 14:00:50.992870 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:50.992835    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj" event={"ID":"fdd31bc5-8505-44f0-922a-6c36f513db0b","Type":"ContainerStarted","Data":"dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f"}
Sep 16 14:00:50.993840 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:50.992895    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj" event={"ID":"fdd31bc5-8505-44f0-922a-6c36f513db0b","Type":"ContainerStarted","Data":"5e22a830d7a0fe945d081652033f35c9cfe30c570826d2fbc26cbafd42015927"}
Sep 16 14:00:50.993840 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:50.993027    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:00:50.995300 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:50.995090    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9" event={"ID":"98067113-7648-45e6-ba4d-52a598f36f01","Type":"ContainerStarted","Data":"9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd"}
Sep 16 14:00:50.995300 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:50.995138    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9" event={"ID":"98067113-7648-45e6-ba4d-52a598f36f01","Type":"ContainerStarted","Data":"74b7de6156775df4dd35b2b2c774172099827d3732f4e4be8b9528e0bf9ef796"}
Sep 16 14:00:50.995300 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:50.995191    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:00:51.014744 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:51.014696    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj" podStartSLOduration=2.014679207 podStartE2EDuration="2.014679207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:51.013407576 +0000 UTC m=+1073.640888951" watchObservedRunningTime="2026-09-16 14:00:51.014679207 +0000 UTC m=+1073.642160582"
Sep 16 14:00:51.034291 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:00:51.034240    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9" podStartSLOduration=2.034228356 podStartE2EDuration="2.034228356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:51.033224336 +0000 UTC m=+1073.660705721" watchObservedRunningTime="2026-09-16 14:00:51.034228356 +0000 UTC m=+1073.661709730"
Sep 16 14:01:02.002714 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:02.002687    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:01:02.003131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:02.002757    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:01:23.342752 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.342713    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"]
Sep 16 14:01:23.347867 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.347849    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.351487 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.351469    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b45798\"/\"kube-root-ca.crt\""
Sep 16 14:01:23.352661 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.352636    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b45798\"/\"default-dockercfg-h87cl\""
Sep 16 14:01:23.352751 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.352648    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b45798\"/\"openshift-service-ca.crt\""
Sep 16 14:01:23.352751 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.352694    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b45798\"/\"noten-model-b45798-kserve-self-signed-certs\""
Sep 16 14:01:23.356215 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.356196    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"]
Sep 16 14:01:23.392455 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.392422    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-model-cache\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.392578 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.392466    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-dshm\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.392578 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.392522    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92pnp\" (UniqueName: \"kubernetes.io/projected/a477a5e4-a517-4c81-a13b-e46e0172e3e9-kube-api-access-92pnp\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.392578 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.392557    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tls-certs\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.392747 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.392604    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tmp-dir\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.392747 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.392671    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-home\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494227 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494199    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-model-cache\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494412 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494240    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-dshm\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494412 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494355    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92pnp\" (UniqueName: \"kubernetes.io/projected/a477a5e4-a517-4c81-a13b-e46e0172e3e9-kube-api-access-92pnp\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494412 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494392    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tls-certs\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494586 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494447    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tmp-dir\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494586 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494476    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-home\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494697 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494600    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-model-cache\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494792 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494770    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tmp-dir\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.494886 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.494864    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-home\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.496393 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.496372    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-dshm\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.496882 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.496863    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tls-certs\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.506667 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.506646    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92pnp\" (UniqueName: \"kubernetes.io/projected/a477a5e4-a517-4c81-a13b-e46e0172e3e9-kube-api-access-92pnp\") pod \"noten-model-b45798-kserve-b8b58979-6lwgk\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") " pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.658432 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.658392    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:23.931975 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:23.931888    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"]
Sep 16 14:01:23.935471 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:01:23.935444    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda477a5e4_a517_4c81_a13b_e46e0172e3e9.slice/crio-d01aeb9baab769dd4e3f072e3a40a945d6890738d099a8f1636df23bdbb3ba51 WatchSource:0}: Error finding container d01aeb9baab769dd4e3f072e3a40a945d6890738d099a8f1636df23bdbb3ba51: Status 404 returned error can't find the container with id d01aeb9baab769dd4e3f072e3a40a945d6890738d099a8f1636df23bdbb3ba51
Sep 16 14:01:24.187768 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:24.187671    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk" event={"ID":"a477a5e4-a517-4c81-a13b-e46e0172e3e9","Type":"ContainerStarted","Data":"67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed"}
Sep 16 14:01:24.187768 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:24.187713    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk" event={"ID":"a477a5e4-a517-4c81-a13b-e46e0172e3e9","Type":"ContainerStarted","Data":"d01aeb9baab769dd4e3f072e3a40a945d6890738d099a8f1636df23bdbb3ba51"}
Sep 16 14:01:24.213268 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:24.213201    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk" podStartSLOduration=1.213182438 podStartE2EDuration="1.213182438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:24.210881495 +0000 UTC m=+1106.838362872" watchObservedRunningTime="2026-09-16 14:01:24.213182438 +0000 UTC m=+1106.840663813"
Sep 16 14:01:33.172175 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.172144    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"]
Sep 16 14:01:33.172669 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.172426    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9" podUID="98067113-7648-45e6-ba4d-52a598f36f01" containerName="payload-processing" containerID="cri-o://9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd" gracePeriod=30
Sep 16 14:01:33.184348 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.182869    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"]
Sep 16 14:01:33.184348 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.183126    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj" podUID="fdd31bc5-8505-44f0-922a-6c36f513db0b" containerName="payload-pre-processing" containerID="cri-o://dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f" gracePeriod=30
Sep 16 14:01:33.475476 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.475451    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:01:33.505793 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.505765    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:01:33.592376 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.592334    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-988lw\" (UniqueName: \"kubernetes.io/projected/fdd31bc5-8505-44f0-922a-6c36f513db0b-kube-api-access-988lw\") pod \"fdd31bc5-8505-44f0-922a-6c36f513db0b\" (UID: \"fdd31bc5-8505-44f0-922a-6c36f513db0b\") "
Sep 16 14:01:33.592376 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.592383    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx775\" (UniqueName: \"kubernetes.io/projected/98067113-7648-45e6-ba4d-52a598f36f01-kube-api-access-wx775\") pod \"98067113-7648-45e6-ba4d-52a598f36f01\" (UID: \"98067113-7648-45e6-ba4d-52a598f36f01\") "
Sep 16 14:01:33.592635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.592415    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd31bc5-8505-44f0-922a-6c36f513db0b-plugins-config-volume\") pod \"fdd31bc5-8505-44f0-922a-6c36f513db0b\" (UID: \"fdd31bc5-8505-44f0-922a-6c36f513db0b\") "
Sep 16 14:01:33.592635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.592442    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98067113-7648-45e6-ba4d-52a598f36f01-plugins-config-volume\") pod \"98067113-7648-45e6-ba4d-52a598f36f01\" (UID: \"98067113-7648-45e6-ba4d-52a598f36f01\") "
Sep 16 14:01:33.592842 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.592819    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd31bc5-8505-44f0-922a-6c36f513db0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdd31bc5-8505-44f0-922a-6c36f513db0b" (UID: "fdd31bc5-8505-44f0-922a-6c36f513db0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:01:33.592917 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.592855    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98067113-7648-45e6-ba4d-52a598f36f01-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98067113-7648-45e6-ba4d-52a598f36f01" (UID: "98067113-7648-45e6-ba4d-52a598f36f01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:01:33.594493 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.594467    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd31bc5-8505-44f0-922a-6c36f513db0b-kube-api-access-988lw" (OuterVolumeSpecName: "kube-api-access-988lw") pod "fdd31bc5-8505-44f0-922a-6c36f513db0b" (UID: "fdd31bc5-8505-44f0-922a-6c36f513db0b"). InnerVolumeSpecName "kube-api-access-988lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:01:33.594634 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.594616    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98067113-7648-45e6-ba4d-52a598f36f01-kube-api-access-wx775" (OuterVolumeSpecName: "kube-api-access-wx775") pod "98067113-7648-45e6-ba4d-52a598f36f01" (UID: "98067113-7648-45e6-ba4d-52a598f36f01"). InnerVolumeSpecName "kube-api-access-wx775". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:01:33.658872 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.658843    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:33.658980 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.658893    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:33.671143 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.671125    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:33.693373 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.693349    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-988lw\" (UniqueName: \"kubernetes.io/projected/fdd31bc5-8505-44f0-922a-6c36f513db0b-kube-api-access-988lw\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:33.693373 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.693372    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx775\" (UniqueName: \"kubernetes.io/projected/98067113-7648-45e6-ba4d-52a598f36f01-kube-api-access-wx775\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:33.693514 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.693381    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd31bc5-8505-44f0-922a-6c36f513db0b-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:33.693514 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:33.693391    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98067113-7648-45e6-ba4d-52a598f36f01-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:34.244186 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.244086    2548 generic.go:356] "Generic (PLEG): container finished" podID="98067113-7648-45e6-ba4d-52a598f36f01" containerID="9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd" exitCode=0
Sep 16 14:01:34.244186 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.244172    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"
Sep 16 14:01:34.244658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.244205    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9" event={"ID":"98067113-7648-45e6-ba4d-52a598f36f01","Type":"ContainerDied","Data":"9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd"}
Sep 16 14:01:34.244658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.244282    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9" event={"ID":"98067113-7648-45e6-ba4d-52a598f36f01","Type":"ContainerDied","Data":"74b7de6156775df4dd35b2b2c774172099827d3732f4e4be8b9528e0bf9ef796"}
Sep 16 14:01:34.244658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.244309    2548 scope.go:117] "RemoveContainer" containerID="9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd"
Sep 16 14:01:34.245797 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.245778    2548 generic.go:356] "Generic (PLEG): container finished" podID="fdd31bc5-8505-44f0-922a-6c36f513db0b" containerID="dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f" exitCode=0
Sep 16 14:01:34.245887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.245823    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj" event={"ID":"fdd31bc5-8505-44f0-922a-6c36f513db0b","Type":"ContainerDied","Data":"dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f"}
Sep 16 14:01:34.245887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.245869    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj" event={"ID":"fdd31bc5-8505-44f0-922a-6c36f513db0b","Type":"ContainerDied","Data":"5e22a830d7a0fe945d081652033f35c9cfe30c570826d2fbc26cbafd42015927"}
Sep 16 14:01:34.245979 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.245918    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"
Sep 16 14:01:34.255282 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.255241    2548 scope.go:117] "RemoveContainer" containerID="9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd"
Sep 16 14:01:34.255530 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:01:34.255512    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd\": container with ID starting with 9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd not found: ID does not exist" containerID="9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd"
Sep 16 14:01:34.255602 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.255536    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd"} err="failed to get container status \"9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd\": rpc error: code = NotFound desc = could not find container \"9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd\": container with ID starting with 9d24ef4578c07bab26d55540da50df645029cbc172e843019143dcc09ac5bebd not found: ID does not exist"
Sep 16 14:01:34.255602 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.255551    2548 scope.go:117] "RemoveContainer" containerID="dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f"
Sep 16 14:01:34.259808 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.259788    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:01:34.263893 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.263875    2548 scope.go:117] "RemoveContainer" containerID="dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f"
Sep 16 14:01:34.264144 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:01:34.264126    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f\": container with ID starting with dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f not found: ID does not exist" containerID="dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f"
Sep 16 14:01:34.264188 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.264149    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f"} err="failed to get container status \"dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f\": rpc error: code = NotFound desc = could not find container \"dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f\": container with ID starting with dddea6e92fc7f216744206d750e2aaefade52f4984ee5e5ce2c83f7312a2e79f not found: ID does not exist"
Sep 16 14:01:34.264857 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.264839    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"]
Sep 16 14:01:34.269854 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.269836    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b5aadeb-5569f845b6-nklbj"]
Sep 16 14:01:34.282299 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.282276    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"]
Sep 16 14:01:34.289211 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:34.289190    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b5aadeb-74866bd45c-tgmb9"]
Sep 16 14:01:35.969992 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:35.969964    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98067113-7648-45e6-ba4d-52a598f36f01" path="/var/lib/kubelet/pods/98067113-7648-45e6-ba4d-52a598f36f01/volumes"
Sep 16 14:01:35.970398 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:35.970383    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd31bc5-8505-44f0-922a-6c36f513db0b" path="/var/lib/kubelet/pods/fdd31bc5-8505-44f0-922a-6c36f513db0b/volumes"
Sep 16 14:01:48.637190 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:48.637158    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"]
Sep 16 14:01:48.750499 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:48.750470    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"]
Sep 16 14:01:48.750677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:48.750656    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s" podUID="621858aa-3e62-41e2-8da9-c6659cee462f" containerName="istio-proxy" containerID="cri-o://1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe" gracePeriod=30
Sep 16 14:01:51.899007 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:51.898973    2548 patch_prober.go:28] interesting pod/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:01:51.899401 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:51.899054    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s" podUID="621858aa-3e62-41e2-8da9-c6659cee462f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:01:54.002625 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.002597    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:01:54.085943 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.085881    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-istio-token\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.085943 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.085913    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-credential-socket\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.085943 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.085934    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621858aa-3e62-41e2-8da9-c6659cee462f-istio-podinfo\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.086322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.085984    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621858aa-3e62-41e2-8da9-c6659cee462f-istiod-ca-cert\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.086322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086011    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-data\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.086322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086081    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-socket\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.086322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086155    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-envoy\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.086322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086211    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-certs\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.086322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086235    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:54.086322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086240    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6swrt\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-kube-api-access-6swrt\") pod \"621858aa-3e62-41e2-8da9-c6659cee462f\" (UID: \"621858aa-3e62-41e2-8da9-c6659cee462f\") "
Sep 16 14:01:54.086645 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086428    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621858aa-3e62-41e2-8da9-c6659cee462f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:01:54.086645 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086500    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-data" (OuterVolumeSpecName: "istio-data") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:54.086645 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086616    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:54.086818 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086689    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:54.086923 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086904    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.086990 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086928    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.086990 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086943    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.086990 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086958    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621858aa-3e62-41e2-8da9-c6659cee462f-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.086990 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.086971    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.088202 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.088171    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-istio-token" (OuterVolumeSpecName: "istio-token") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:01:54.088620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.088600    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/621858aa-3e62-41e2-8da9-c6659cee462f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:01:54.088782 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.088762    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-kube-api-access-6swrt" (OuterVolumeSpecName: "kube-api-access-6swrt") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "kube-api-access-6swrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:01:54.088879 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.088854    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "621858aa-3e62-41e2-8da9-c6659cee462f" (UID: "621858aa-3e62-41e2-8da9-c6659cee462f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:54.188016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.187982    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621858aa-3e62-41e2-8da9-c6659cee462f-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.188016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.188019    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6swrt\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-kube-api-access-6swrt\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.188216 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.188034    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621858aa-3e62-41e2-8da9-c6659cee462f-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.188216 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.188043    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621858aa-3e62-41e2-8da9-c6659cee462f-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:01:54.371254 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.371224    2548 generic.go:356] "Generic (PLEG): container finished" podID="621858aa-3e62-41e2-8da9-c6659cee462f" containerID="1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe" exitCode=0
Sep 16 14:01:54.371391 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.371285    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"
Sep 16 14:01:54.371391 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.371302    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s" event={"ID":"621858aa-3e62-41e2-8da9-c6659cee462f","Type":"ContainerDied","Data":"1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe"}
Sep 16 14:01:54.371391 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.371358    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s" event={"ID":"621858aa-3e62-41e2-8da9-c6659cee462f","Type":"ContainerDied","Data":"00649a74a4b4e9d7badeff5ad877e6b94a82b9bea0bbc76cf1b0cbd605687149"}
Sep 16 14:01:54.371391 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.371382    2548 scope.go:117] "RemoveContainer" containerID="1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe"
Sep 16 14:01:54.381473 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.381452    2548 scope.go:117] "RemoveContainer" containerID="1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe"
Sep 16 14:01:54.381765 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:01:54.381746    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe\": container with ID starting with 1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe not found: ID does not exist" containerID="1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe"
Sep 16 14:01:54.381840 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.381772    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe"} err="failed to get container status \"1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe\": rpc error: code = NotFound desc = could not find container \"1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe\": container with ID starting with 1c10d1ad75b15010ce56b61895f73ce5c00910b7b92b438aa325e0bdc38620fe not found: ID does not exist"
Sep 16 14:01:54.395763 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.395742    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"]
Sep 16 14:01:54.406366 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:54.406346    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-b45798-openshift-default-7f54f884b-4hq9s"]
Sep 16 14:01:55.971279 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:55.971232    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="621858aa-3e62-41e2-8da9-c6659cee462f" path="/var/lib/kubelet/pods/621858aa-3e62-41e2-8da9-c6659cee462f/volumes"
Sep 16 14:01:56.229622 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.229541    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"]
Sep 16 14:01:56.229950 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.229937    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd31bc5-8505-44f0-922a-6c36f513db0b" containerName="payload-pre-processing"
Sep 16 14:01:56.230016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.229950    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd31bc5-8505-44f0-922a-6c36f513db0b" containerName="payload-pre-processing"
Sep 16 14:01:56.230016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.229964    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98067113-7648-45e6-ba4d-52a598f36f01" containerName="payload-processing"
Sep 16 14:01:56.230016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.229969    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="98067113-7648-45e6-ba4d-52a598f36f01" containerName="payload-processing"
Sep 16 14:01:56.230016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.229982    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="621858aa-3e62-41e2-8da9-c6659cee462f" containerName="istio-proxy"
Sep 16 14:01:56.230016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.229988    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="621858aa-3e62-41e2-8da9-c6659cee462f" containerName="istio-proxy"
Sep 16 14:01:56.230297 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.230045    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd31bc5-8505-44f0-922a-6c36f513db0b" containerName="payload-pre-processing"
Sep 16 14:01:56.230297 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.230054    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="621858aa-3e62-41e2-8da9-c6659cee462f" containerName="istio-proxy"
Sep 16 14:01:56.230297 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.230060    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="98067113-7648-45e6-ba4d-52a598f36f01" containerName="payload-processing"
Sep 16 14:01:56.233270 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.233238    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.237957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.237939    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-740a25-openshift-default-dockercfg-m4bbp\""
Sep 16 14:01:56.254038 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.254010    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"]
Sep 16 14:01:56.411652 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.411616    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-data\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.411845 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.411678    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-socket\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.411845 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.411744    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96eb6564-246d-46fb-b204-8fdff138a99b-istio-podinfo\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.411845 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.411796    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5f4n\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-kube-api-access-d5f4n\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.411845 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.411824    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-credential-socket\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.412012 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.411902    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-certs\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.412012 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.411932    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96eb6564-246d-46fb-b204-8fdff138a99b-istiod-ca-cert\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.412012 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.411974    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-envoy\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.412147 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.412045    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-istio-token\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.512825 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.512748    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96eb6564-246d-46fb-b204-8fdff138a99b-istio-podinfo\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.512825 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.512786    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5f4n\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-kube-api-access-d5f4n\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513015 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.512847    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-credential-socket\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513015 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.512897    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-certs\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513015 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.512917    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96eb6564-246d-46fb-b204-8fdff138a99b-istiod-ca-cert\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513015 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.512953    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-envoy\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513193 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.513066    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-istio-token\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513193 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.513125    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-data\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513193 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.513175    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-socket\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513385 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.513337    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-credential-socket\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513446 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.513383    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-certs\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513599 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.513573    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-socket\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513798 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.513777    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-data\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.513979 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.513959    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96eb6564-246d-46fb-b204-8fdff138a99b-istiod-ca-cert\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.515439 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.515420    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96eb6564-246d-46fb-b204-8fdff138a99b-istio-podinfo\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.515439 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.515430    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-envoy\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.522083 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.522052    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-istio-token\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.522180 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.522144    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5f4n\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-kube-api-access-d5f4n\") pod \"e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") " pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.543113 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.543091    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:56.825436 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.825409    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"]
Sep 16 14:01:56.827405 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:01:56.827375    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96eb6564_246d_46fb_b204_8fdff138a99b.slice/crio-cf9a1e17cc3f71e9d430e07db0a785fb13d62a5c746d17cd1197089030f3f31c WatchSource:0}: Error finding container cf9a1e17cc3f71e9d430e07db0a785fb13d62a5c746d17cd1197089030f3f31c: Status 404 returned error can't find the container with id cf9a1e17cc3f71e9d430e07db0a785fb13d62a5c746d17cd1197089030f3f31c
Sep 16 14:01:56.829753 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.829719    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:01:56.829835 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.829785    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:01:56.829835 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:56.829813    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:01:57.391507 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:57.391472    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" event={"ID":"96eb6564-246d-46fb-b204-8fdff138a99b","Type":"ContainerStarted","Data":"33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3"}
Sep 16 14:01:57.391984 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:57.391534    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" event={"ID":"96eb6564-246d-46fb-b204-8fdff138a99b","Type":"ContainerStarted","Data":"cf9a1e17cc3f71e9d430e07db0a785fb13d62a5c746d17cd1197089030f3f31c"}
Sep 16 14:01:57.441468 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:57.441416    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" podStartSLOduration=1.441401466 podStartE2EDuration="1.441401466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:57.437241645 +0000 UTC m=+1140.064723044" watchObservedRunningTime="2026-09-16 14:01:57.441401466 +0000 UTC m=+1140.068882899"
Sep 16 14:01:57.543748 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:57.543719    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:57.545228 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:57.545205    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:01:57.545346 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:57.545275    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 14:01:58.547459 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:58.547427    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:59.402564 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:59.402531    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:01:59.403520 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:01:59.403500    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:02:03.347126 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.347092    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"]
Sep 16 14:02:03.656097 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.656026    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk" podUID="a477a5e4-a517-4c81-a13b-e46e0172e3e9" containerName="main" containerID="cri-o://67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed" gracePeriod=45
Sep 16 14:02:03.922587 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.922518    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:02:03.990855 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.990722    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92pnp\" (UniqueName: \"kubernetes.io/projected/a477a5e4-a517-4c81-a13b-e46e0172e3e9-kube-api-access-92pnp\") pod \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") "
Sep 16 14:02:03.990855 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.990775    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tmp-dir\") pod \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") "
Sep 16 14:02:03.990855 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.990819    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tls-certs\") pod \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") "
Sep 16 14:02:03.991288 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.990864    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-dshm\") pod \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") "
Sep 16 14:02:03.991288 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.990969    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-home\") pod \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") "
Sep 16 14:02:03.991288 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.991008    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-model-cache\") pod \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\" (UID: \"a477a5e4-a517-4c81-a13b-e46e0172e3e9\") "
Sep 16 14:02:03.992409 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.992186    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a477a5e4-a517-4c81-a13b-e46e0172e3e9" (UID: "a477a5e4-a517-4c81-a13b-e46e0172e3e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:03.992683 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.992529    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a477a5e4-a517-4c81-a13b-e46e0172e3e9" (UID: "a477a5e4-a517-4c81-a13b-e46e0172e3e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:03.992843 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.992811    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-home" (OuterVolumeSpecName: "home") pod "a477a5e4-a517-4c81-a13b-e46e0172e3e9" (UID: "a477a5e4-a517-4c81-a13b-e46e0172e3e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:03.994933 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.994908    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-dshm" (OuterVolumeSpecName: "dshm") pod "a477a5e4-a517-4c81-a13b-e46e0172e3e9" (UID: "a477a5e4-a517-4c81-a13b-e46e0172e3e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:03.995182 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.995139    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a477a5e4-a517-4c81-a13b-e46e0172e3e9" (UID: "a477a5e4-a517-4c81-a13b-e46e0172e3e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:02:03.995794 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:03.995766    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a477a5e4-a517-4c81-a13b-e46e0172e3e9-kube-api-access-92pnp" (OuterVolumeSpecName: "kube-api-access-92pnp") pod "a477a5e4-a517-4c81-a13b-e46e0172e3e9" (UID: "a477a5e4-a517-4c81-a13b-e46e0172e3e9"). InnerVolumeSpecName "kube-api-access-92pnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:04.092338 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.092308    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:04.092338 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.092334    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92pnp\" (UniqueName: \"kubernetes.io/projected/a477a5e4-a517-4c81-a13b-e46e0172e3e9-kube-api-access-92pnp\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:04.092338 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.092344    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:04.092338 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.092354    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a477a5e4-a517-4c81-a13b-e46e0172e3e9-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:04.092600 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.092362    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:04.092600 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.092372    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a477a5e4-a517-4c81-a13b-e46e0172e3e9-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:04.429049 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.429015    2548 generic.go:356] "Generic (PLEG): container finished" podID="a477a5e4-a517-4c81-a13b-e46e0172e3e9" containerID="67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed" exitCode=0
Sep 16 14:02:04.429464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.429074    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"
Sep 16 14:02:04.429464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.429101    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk" event={"ID":"a477a5e4-a517-4c81-a13b-e46e0172e3e9","Type":"ContainerDied","Data":"67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed"}
Sep 16 14:02:04.429464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.429149    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk" event={"ID":"a477a5e4-a517-4c81-a13b-e46e0172e3e9","Type":"ContainerDied","Data":"d01aeb9baab769dd4e3f072e3a40a945d6890738d099a8f1636df23bdbb3ba51"}
Sep 16 14:02:04.429464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.429163    2548 scope.go:117] "RemoveContainer" containerID="67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed"
Sep 16 14:02:04.437845 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.437826    2548 scope.go:117] "RemoveContainer" containerID="67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed"
Sep 16 14:02:04.438121 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:02:04.438097    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed\": container with ID starting with 67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed not found: ID does not exist" containerID="67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed"
Sep 16 14:02:04.438197 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.438129    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed"} err="failed to get container status \"67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed\": rpc error: code = NotFound desc = could not find container \"67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed\": container with ID starting with 67e7f7d6b1807f7a94545f06a5bd3ff89620f1b2282dba0f59ccb9e0dca380ed not found: ID does not exist"
Sep 16 14:02:04.461070 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.461038    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"]
Sep 16 14:02:04.464893 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.464866    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-b45798/noten-model-b45798-kserve-b8b58979-6lwgk"]
Sep 16 14:02:04.789074 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.789001    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"]
Sep 16 14:02:04.789482 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.789467    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a477a5e4-a517-4c81-a13b-e46e0172e3e9" containerName="main"
Sep 16 14:02:04.789482 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.789483    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="a477a5e4-a517-4c81-a13b-e46e0172e3e9" containerName="main"
Sep 16 14:02:04.789581 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.789558    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="a477a5e4-a517-4c81-a13b-e46e0172e3e9" containerName="main"
Sep 16 14:02:04.794131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.794110    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:04.796712 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.796690    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-740a25-dockercfg-wkpz8\""
Sep 16 14:02:04.796828 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.796728    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-740a25\""
Sep 16 14:02:04.807225 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.807202    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"]
Sep 16 14:02:04.899233 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.899200    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098a8c2e-e69c-4432-b1be-f56a3529e5c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2\" (UID: \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:04.899423 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:04.899271    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxth\" (UniqueName: \"kubernetes.io/projected/098a8c2e-e69c-4432-b1be-f56a3529e5c9-kube-api-access-djxth\") pod \"payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2\" (UID: \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:05.000462 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:05.000420    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098a8c2e-e69c-4432-b1be-f56a3529e5c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2\" (UID: \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:05.000632 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:05.000509    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxth\" (UniqueName: \"kubernetes.io/projected/098a8c2e-e69c-4432-b1be-f56a3529e5c9-kube-api-access-djxth\") pod \"payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2\" (UID: \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:05.001111 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:05.001091    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098a8c2e-e69c-4432-b1be-f56a3529e5c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2\" (UID: \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:05.011124 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:05.011107    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxth\" (UniqueName: \"kubernetes.io/projected/098a8c2e-e69c-4432-b1be-f56a3529e5c9-kube-api-access-djxth\") pod \"payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2\" (UID: \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:05.104329 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:05.104241    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:05.377165 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:05.377133    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"]
Sep 16 14:02:05.378962 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:02:05.378936    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098a8c2e_e69c_4432_b1be_f56a3529e5c9.slice/crio-be626c1dde57473a8652ad2c0c5efaeca065126cf537fbda0f635a7e943c5bde WatchSource:0}: Error finding container be626c1dde57473a8652ad2c0c5efaeca065126cf537fbda0f635a7e943c5bde: Status 404 returned error can't find the container with id be626c1dde57473a8652ad2c0c5efaeca065126cf537fbda0f635a7e943c5bde
Sep 16 14:02:05.436131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:05.436103    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2" event={"ID":"098a8c2e-e69c-4432-b1be-f56a3529e5c9","Type":"ContainerStarted","Data":"be626c1dde57473a8652ad2c0c5efaeca065126cf537fbda0f635a7e943c5bde"}
Sep 16 14:02:05.970203 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:05.970174    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a477a5e4-a517-4c81-a13b-e46e0172e3e9" path="/var/lib/kubelet/pods/a477a5e4-a517-4c81-a13b-e46e0172e3e9/volumes"
Sep 16 14:02:06.443764 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:06.443729    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2" event={"ID":"098a8c2e-e69c-4432-b1be-f56a3529e5c9","Type":"ContainerStarted","Data":"35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8"}
Sep 16 14:02:06.444140 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:06.443800    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:06.464318 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:06.464270    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2" podStartSLOduration=2.464238103 podStartE2EDuration="2.464238103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:06.46166703 +0000 UTC m=+1149.089148404" watchObservedRunningTime="2026-09-16 14:02:06.464238103 +0000 UTC m=+1149.091719883"
Sep 16 14:02:14.615871 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:14.615835    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"]
Sep 16 14:02:14.617760 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:14.617738    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b52a89-29826120-ckh4v"]
Sep 16 14:02:14.649180 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:14.649150    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"]
Sep 16 14:02:14.649463 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:14.649431    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt" podUID="727b3305-af44-4e6b-ad30-bb17cbf3def0" containerName="payload-processing" containerID="cri-o://27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95" gracePeriod=30
Sep 16 14:02:14.935792 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:14.935762    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 14:02:15.006272 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.005580    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b3305-af44-4e6b-ad30-bb17cbf3def0-plugins-config-volume\") pod \"727b3305-af44-4e6b-ad30-bb17cbf3def0\" (UID: \"727b3305-af44-4e6b-ad30-bb17cbf3def0\") "
Sep 16 14:02:15.006272 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.005715    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qgmm\" (UniqueName: \"kubernetes.io/projected/727b3305-af44-4e6b-ad30-bb17cbf3def0-kube-api-access-4qgmm\") pod \"727b3305-af44-4e6b-ad30-bb17cbf3def0\" (UID: \"727b3305-af44-4e6b-ad30-bb17cbf3def0\") "
Sep 16 14:02:15.010271 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.006592    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727b3305-af44-4e6b-ad30-bb17cbf3def0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "727b3305-af44-4e6b-ad30-bb17cbf3def0" (UID: "727b3305-af44-4e6b-ad30-bb17cbf3def0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:02:15.028261 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.027969    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727b3305-af44-4e6b-ad30-bb17cbf3def0-kube-api-access-4qgmm" (OuterVolumeSpecName: "kube-api-access-4qgmm") pod "727b3305-af44-4e6b-ad30-bb17cbf3def0" (UID: "727b3305-af44-4e6b-ad30-bb17cbf3def0"). InnerVolumeSpecName "kube-api-access-4qgmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:15.106579 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.106549    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b3305-af44-4e6b-ad30-bb17cbf3def0-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:15.106579 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.106575    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qgmm\" (UniqueName: \"kubernetes.io/projected/727b3305-af44-4e6b-ad30-bb17cbf3def0-kube-api-access-4qgmm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:15.501143 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.501098    2548 generic.go:356] "Generic (PLEG): container finished" podID="727b3305-af44-4e6b-ad30-bb17cbf3def0" containerID="27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95" exitCode=0
Sep 16 14:02:15.501358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.501168    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt" event={"ID":"727b3305-af44-4e6b-ad30-bb17cbf3def0","Type":"ContainerDied","Data":"27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95"}
Sep 16 14:02:15.501358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.501217    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt" event={"ID":"727b3305-af44-4e6b-ad30-bb17cbf3def0","Type":"ContainerDied","Data":"237b9f725a0f937e6144d44a6441b27c1b57b616858027db4c23a8140d343e74"}
Sep 16 14:02:15.501358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.501265    2548 scope.go:117] "RemoveContainer" containerID="27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95"
Sep 16 14:02:15.501558 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.501426    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"
Sep 16 14:02:15.513598 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.513574    2548 scope.go:117] "RemoveContainer" containerID="27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95"
Sep 16 14:02:15.513909 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:02:15.513888    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95\": container with ID starting with 27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95 not found: ID does not exist" containerID="27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95"
Sep 16 14:02:15.513953 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.513918    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95"} err="failed to get container status \"27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95\": rpc error: code = NotFound desc = could not find container \"27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95\": container with ID starting with 27d604ae75c24516a4e9bafbfe768e25f974bf3553d99174b4692f17a6c51a95 not found: ID does not exist"
Sep 16 14:02:15.527686 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.527661    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"]
Sep 16 14:02:15.529819 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.529795    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b52a89-c7cc6ddc9-xcpdt"]
Sep 16 14:02:15.970349 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.970314    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50efba3a-ebba-447b-8a86-f629209a63b7" path="/var/lib/kubelet/pods/50efba3a-ebba-447b-8a86-f629209a63b7/volumes"
Sep 16 14:02:15.970840 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:15.970812    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727b3305-af44-4e6b-ad30-bb17cbf3def0" path="/var/lib/kubelet/pods/727b3305-af44-4e6b-ad30-bb17cbf3def0/volumes"
Sep 16 14:02:17.452309 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:17.452279    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:02:18.365842 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.365794    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf" podUID="8eb68f45-7a1f-45b9-a71c-5465ee4153bd" containerName="main" containerID="cri-o://c226a902bde28c1d9be0d81b292032eab312650046c4976fff6f48afb631ad78" gracePeriod=45
Sep 16 14:02:18.527626 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.527597    2548 generic.go:356] "Generic (PLEG): container finished" podID="8eb68f45-7a1f-45b9-a71c-5465ee4153bd" containerID="c226a902bde28c1d9be0d81b292032eab312650046c4976fff6f48afb631ad78" exitCode=0
Sep 16 14:02:18.528007 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.527672    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf" event={"ID":"8eb68f45-7a1f-45b9-a71c-5465ee4153bd","Type":"ContainerDied","Data":"c226a902bde28c1d9be0d81b292032eab312650046c4976fff6f48afb631ad78"}
Sep 16 14:02:18.654897 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.654872    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 14:02:18.741111 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741084    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tls-certs\") pod \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") "
Sep 16 14:02:18.741275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741147    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-model-cache\") pod \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") "
Sep 16 14:02:18.741275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741172    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-home\") pod \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") "
Sep 16 14:02:18.741275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741219    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4px2\" (UniqueName: \"kubernetes.io/projected/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-kube-api-access-p4px2\") pod \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") "
Sep 16 14:02:18.741275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741263    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tmp-dir\") pod \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") "
Sep 16 14:02:18.741506 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741321    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-dshm\") pod \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\" (UID: \"8eb68f45-7a1f-45b9-a71c-5465ee4153bd\") "
Sep 16 14:02:18.741506 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741440    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-home" (OuterVolumeSpecName: "home") pod "8eb68f45-7a1f-45b9-a71c-5465ee4153bd" (UID: "8eb68f45-7a1f-45b9-a71c-5465ee4153bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:18.741609 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741549    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8eb68f45-7a1f-45b9-a71c-5465ee4153bd" (UID: "8eb68f45-7a1f-45b9-a71c-5465ee4153bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:18.741782 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741708    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:18.741782 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.741757    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:18.742059 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.742034    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "8eb68f45-7a1f-45b9-a71c-5465ee4153bd" (UID: "8eb68f45-7a1f-45b9-a71c-5465ee4153bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:18.743518 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.743495    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-dshm" (OuterVolumeSpecName: "dshm") pod "8eb68f45-7a1f-45b9-a71c-5465ee4153bd" (UID: "8eb68f45-7a1f-45b9-a71c-5465ee4153bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:18.743600 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.743528    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-kube-api-access-p4px2" (OuterVolumeSpecName: "kube-api-access-p4px2") pod "8eb68f45-7a1f-45b9-a71c-5465ee4153bd" (UID: "8eb68f45-7a1f-45b9-a71c-5465ee4153bd"). InnerVolumeSpecName "kube-api-access-p4px2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:18.743600 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.743564    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8eb68f45-7a1f-45b9-a71c-5465ee4153bd" (UID: "8eb68f45-7a1f-45b9-a71c-5465ee4153bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:02:18.842677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.842633    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4px2\" (UniqueName: \"kubernetes.io/projected/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-kube-api-access-p4px2\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:18.842677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.842672    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:18.842677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.842683    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:18.842677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:18.842694    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb68f45-7a1f-45b9-a71c-5465ee4153bd-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:19.534830 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:19.534796    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf" event={"ID":"8eb68f45-7a1f-45b9-a71c-5465ee4153bd","Type":"ContainerDied","Data":"b581b1e4c656b7ef5c7b9122f86ee90fbaaf30cb9d683f9a4bc182104cdb8fb6"}
Sep 16 14:02:19.534830 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:19.534820    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"
Sep 16 14:02:19.535388 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:19.534849    2548 scope.go:117] "RemoveContainer" containerID="c226a902bde28c1d9be0d81b292032eab312650046c4976fff6f48afb631ad78"
Sep 16 14:02:19.563739 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:19.563709    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"]
Sep 16 14:02:19.579181 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:19.579149    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-95743e/test-model-95743e-kserve-7cf8988bc5-j2bqf"]
Sep 16 14:02:19.970732 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:19.970696    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eb68f45-7a1f-45b9-a71c-5465ee4153bd" path="/var/lib/kubelet/pods/8eb68f45-7a1f-45b9-a71c-5465ee4153bd/volumes"
Sep 16 14:02:22.530574 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.530543    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"]
Sep 16 14:02:22.531087 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.530734    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" containerID="cri-o://c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd" gracePeriod=30
Sep 16 14:02:22.555339 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.555315    2548 generic.go:356] "Generic (PLEG): container finished" podID="96eb6564-246d-46fb-b204-8fdff138a99b" containerID="33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3" exitCode=0
Sep 16 14:02:22.555463 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.555393    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" event={"ID":"96eb6564-246d-46fb-b204-8fdff138a99b","Type":"ContainerDied","Data":"33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3"}
Sep 16 14:02:22.555848 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.555830    2548 scope.go:117] "RemoveContainer" containerID="33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3"
Sep 16 14:02:22.556945 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.556912    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:22.557025 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.556976    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:22.557025 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.557011    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:22.871974 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.871942    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"]
Sep 16 14:02:22.872438 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.872424    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727b3305-af44-4e6b-ad30-bb17cbf3def0" containerName="payload-processing"
Sep 16 14:02:22.872488 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.872440    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="727b3305-af44-4e6b-ad30-bb17cbf3def0" containerName="payload-processing"
Sep 16 14:02:22.872488 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.872454    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb68f45-7a1f-45b9-a71c-5465ee4153bd" containerName="main"
Sep 16 14:02:22.872488 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.872460    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb68f45-7a1f-45b9-a71c-5465ee4153bd" containerName="main"
Sep 16 14:02:22.872592 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.872516    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb68f45-7a1f-45b9-a71c-5465ee4153bd" containerName="main"
Sep 16 14:02:22.872592 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.872524    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="727b3305-af44-4e6b-ad30-bb17cbf3def0" containerName="payload-processing"
Sep 16 14:02:22.875040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.875024    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"
Sep 16 14:02:22.881369 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.881346    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:02:22.881691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.881637    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:02:22.892524 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.892502    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"]
Sep 16 14:02:22.986778 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:22.986747    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44835-5514-4aee-b989-9626d54318c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95743e-2lk99\" (UID: \"0cf44835-5514-4aee-b989-9626d54318c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"
Sep 16 14:02:23.088722 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:23.088670    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44835-5514-4aee-b989-9626d54318c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95743e-2lk99\" (UID: \"0cf44835-5514-4aee-b989-9626d54318c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"
Sep 16 14:02:23.089881 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:23.089399    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44835-5514-4aee-b989-9626d54318c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-95743e-2lk99\" (UID: \"0cf44835-5514-4aee-b989-9626d54318c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"
Sep 16 14:02:23.185423 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:23.185317    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"
Sep 16 14:02:23.472040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:23.471064    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"]
Sep 16 14:02:23.562505 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:23.562470    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99" event={"ID":"0cf44835-5514-4aee-b989-9626d54318c3","Type":"ContainerStarted","Data":"927c07a1f7c164304152fd61da501b6b763a7b702ac9c9a4ba140639178fb751"}
Sep 16 14:02:23.564419 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:23.564397    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" event={"ID":"96eb6564-246d-46fb-b204-8fdff138a99b","Type":"ContainerStarted","Data":"b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3"}
Sep 16 14:02:24.543877 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:24.543838    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:02:24.544125 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:24.544102    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:02:24.544191 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:24.544155    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 14:02:24.571521 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:24.571494    2548 generic.go:356] "Generic (PLEG): container finished" podID="0cf44835-5514-4aee-b989-9626d54318c3" containerID="adcf54b8dc521efdd5d3b782791bde69a67c82f97de1b5c021c154a0ad45efe1" exitCode=0
Sep 16 14:02:24.571842 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:24.571577    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99" event={"ID":"0cf44835-5514-4aee-b989-9626d54318c3","Type":"ContainerDied","Data":"adcf54b8dc521efdd5d3b782791bde69a67c82f97de1b5c021c154a0ad45efe1"}
Sep 16 14:02:24.995880 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:24.995847    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"]
Sep 16 14:02:25.009510 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.009485    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.014264 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.013781    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9187f5-openshift-default-dockercfg-st6x7\""
Sep 16 14:02:25.022518 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.022497    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"]
Sep 16 14:02:25.120262 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.120194    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-envoy\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.120613 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.120592    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-certs\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.120828 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.120809    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-credential-socket\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.121027 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.120982    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-socket\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.121161 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.121146    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-data\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.121391 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.121375    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0117d88d-499d-4196-badf-6e77bd473b17-istiod-ca-cert\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.121601 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.121584    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0117d88d-499d-4196-badf-6e77bd473b17-istio-podinfo\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.121773 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.121749    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcr2k\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-kube-api-access-fcr2k\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.121917 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.121903    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-istio-token\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223268 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223215    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0117d88d-499d-4196-badf-6e77bd473b17-istiod-ca-cert\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223288    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0117d88d-499d-4196-badf-6e77bd473b17-istio-podinfo\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223318    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcr2k\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-kube-api-access-fcr2k\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223340    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-istio-token\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223374    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-envoy\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223428    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-certs\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223743 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223471    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-credential-socket\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223743 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223506    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-socket\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223743 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223533    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-data\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.223987 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223960    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-data\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.224057 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.223970    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-socket\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.224057 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.224047    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-certs\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.224165 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.224099    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0117d88d-499d-4196-badf-6e77bd473b17-istiod-ca-cert\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.224230 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.224209    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-credential-socket\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.225846 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.225817    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-envoy\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.225948 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.225887    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0117d88d-499d-4196-badf-6e77bd473b17-istio-podinfo\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.239658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.239629    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-istio-token\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.239977 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.239954    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcr2k\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-kube-api-access-fcr2k\") pod \"e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") " pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.326823 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.326738    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:25.544014 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.543973    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:02:25.544217 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.544034    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 14:02:25.622332 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.622306    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"]
Sep 16 14:02:25.624942 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:02:25.624919    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0117d88d_499d_4196_badf_6e77bd473b17.slice/crio-59cbf577def504a523a6c588b1d087785e923d3fee7b81490a5493f1d5940804 WatchSource:0}: Error finding container 59cbf577def504a523a6c588b1d087785e923d3fee7b81490a5493f1d5940804: Status 404 returned error can't find the container with id 59cbf577def504a523a6c588b1d087785e923d3fee7b81490a5493f1d5940804
Sep 16 14:02:25.628592 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.628558    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:25.628670 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.628640    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:25.628711 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.628681    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:25.745024 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.745001    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"
Sep 16 14:02:25.830544 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.830508    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44835-5514-4aee-b989-9626d54318c3-maas-api-service-ca\") pod \"0cf44835-5514-4aee-b989-9626d54318c3\" (UID: \"0cf44835-5514-4aee-b989-9626d54318c3\") "
Sep 16 14:02:25.831125 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.831047    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf44835-5514-4aee-b989-9626d54318c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cf44835-5514-4aee-b989-9626d54318c3" (UID: "0cf44835-5514-4aee-b989-9626d54318c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:02:25.931533 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:25.931498    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44835-5514-4aee-b989-9626d54318c3-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:26.544234 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.544153    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:02:26.544234 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.544205    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 14:02:26.592943 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.592906    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"
Sep 16 14:02:26.593128 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.592942    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99" event={"ID":"0cf44835-5514-4aee-b989-9626d54318c3","Type":"ContainerDied","Data":"927c07a1f7c164304152fd61da501b6b763a7b702ac9c9a4ba140639178fb751"}
Sep 16 14:02:26.593128 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.593017    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="927c07a1f7c164304152fd61da501b6b763a7b702ac9c9a4ba140639178fb751"
Sep 16 14:02:26.595813 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.595774    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" event={"ID":"0117d88d-499d-4196-badf-6e77bd473b17","Type":"ContainerStarted","Data":"1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0"}
Sep 16 14:02:26.595944 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.595834    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" event={"ID":"0117d88d-499d-4196-badf-6e77bd473b17","Type":"ContainerStarted","Data":"59cbf577def504a523a6c588b1d087785e923d3fee7b81490a5493f1d5940804"}
Sep 16 14:02:26.620075 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.620024    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" podStartSLOduration=2.620009714 podStartE2EDuration="2.620009714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:26.619948196 +0000 UTC m=+1169.247429575" watchObservedRunningTime="2026-09-16 14:02:26.620009714 +0000 UTC m=+1169.247491088"
Sep 16 14:02:26.657712 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.657683    2548 patch_prober.go:28] interesting pod/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:02:26.658055 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:26.657732    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:02:27.327113 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.327074    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:27.332171 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.332147    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:27.544340 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.544306    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 14:02:27.544460 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.544365    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 14:02:27.604236 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.604151    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:27.605882 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.605858    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:02:27.830631 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.830606    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 14:02:27.951398 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951363    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-certs\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951594 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951421    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-data\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951594 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951449    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-token\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951594 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951487    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-credential-socket\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951594 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951588    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-envoy\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951832 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951629    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncvd4\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-kube-api-access-ncvd4\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951832 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951660    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-socket\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951832 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951688    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istiod-ca-cert\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951832 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951688    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:27.951832 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951704    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-data" (OuterVolumeSpecName: "istio-data") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:27.951832 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951755    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-podinfo\") pod \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\" (UID: \"d085667e-8ce6-402b-9d65-9f3ebf25abbb\") "
Sep 16 14:02:27.951832 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951789    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:27.952196 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.951962    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:27.952196 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.952041    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:02:27.952196 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.952189    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:27.952485 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.952208    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:27.952485 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.952221    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:27.952485 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.952233    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:27.952485 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.952261    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:27.965701 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.965672    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:02:27.965790 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.965765    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-token" (OuterVolumeSpecName: "istio-token") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:27.965882 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.965786    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-kube-api-access-ncvd4" (OuterVolumeSpecName: "kube-api-access-ncvd4") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "kube-api-access-ncvd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:27.968044 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.967041    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d085667e-8ce6-402b-9d65-9f3ebf25abbb" (UID: "d085667e-8ce6-402b-9d65-9f3ebf25abbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:27.974127 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.974109    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"]
Sep 16 14:02:27.974697 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.974680    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy"
Sep 16 14:02:27.974753 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.974701    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy"
Sep 16 14:02:27.974753 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.974733    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf44835-5514-4aee-b989-9626d54318c3" containerName="revoke-keys"
Sep 16 14:02:27.974753 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.974742    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf44835-5514-4aee-b989-9626d54318c3" containerName="revoke-keys"
Sep 16 14:02:27.974867 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.974845    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy"
Sep 16 14:02:27.974867 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.974856    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf44835-5514-4aee-b989-9626d54318c3" containerName="revoke-keys"
Sep 16 14:02:27.974867 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.974865    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy"
Sep 16 14:02:27.986981 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.986475    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:27.996508 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.996484    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"openshift-service-ca.crt\""
Sep 16 14:02:27.997687 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.996771    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"kube-root-ca.crt\""
Sep 16 14:02:27.997687 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.996969    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"default-dockercfg-vk8k8\""
Sep 16 14:02:27.997687 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:27.997403    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"distinct-2-w0-035c64-kserve-self-signed-certs\""
Sep 16 14:02:28.012722 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.010827    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"]
Sep 16 14:02:28.053786 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.053758    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:28.053999 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.053982    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:28.054109 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.054095    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d085667e-8ce6-402b-9d65-9f3ebf25abbb-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:28.054204 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.054192    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvd4\" (UniqueName: \"kubernetes.io/projected/d085667e-8ce6-402b-9d65-9f3ebf25abbb-kube-api-access-ncvd4\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:28.155209 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.155178    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrjrr\" (UniqueName: \"kubernetes.io/projected/eed57543-ec93-4351-8f3f-5c4d5a33bf98-kube-api-access-rrjrr\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.155396 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.155235    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tls-certs\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.155396 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.155288    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tmp-dir\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.155396 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.155335    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-home\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.155562 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.155394    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-model-cache\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.155562 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.155445    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-dshm\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.256659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.256570    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrjrr\" (UniqueName: \"kubernetes.io/projected/eed57543-ec93-4351-8f3f-5c4d5a33bf98-kube-api-access-rrjrr\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.256659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.256638    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tls-certs\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.256934 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.256663    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tmp-dir\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.256934 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.256699    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-home\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.256934 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.256727    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-model-cache\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.256934 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.256849    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-dshm\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.257147 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.257011    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-model-cache\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.257147 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.257068    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-home\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.257147 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.257126    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tmp-dir\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.258963 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.258932    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-dshm\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.259155 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.259131    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tls-certs\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.279824 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.279794    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrjrr\" (UniqueName: \"kubernetes.io/projected/eed57543-ec93-4351-8f3f-5c4d5a33bf98-kube-api-access-rrjrr\") pod \"distinct-2-w0-035c64-kserve-5f67db767-p8n75\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") " pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.319791 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.319748    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:28.595414 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.595384    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"]
Sep 16 14:02:28.597281 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:02:28.597214    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeed57543_ec93_4351_8f3f_5c4d5a33bf98.slice/crio-3715f25335d92fe6f8c1569409f424c3858c10e4ca1349e95fa8575ffda3a4c1 WatchSource:0}: Error finding container 3715f25335d92fe6f8c1569409f424c3858c10e4ca1349e95fa8575ffda3a4c1: Status 404 returned error can't find the container with id 3715f25335d92fe6f8c1569409f424c3858c10e4ca1349e95fa8575ffda3a4c1
Sep 16 14:02:28.611600 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.611578    2548 generic.go:356] "Generic (PLEG): container finished" podID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerID="c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd" exitCode=0
Sep 16 14:02:28.611715 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.611636    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"
Sep 16 14:02:28.611715 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.611658    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" event={"ID":"d085667e-8ce6-402b-9d65-9f3ebf25abbb","Type":"ContainerDied","Data":"c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd"}
Sep 16 14:02:28.611826 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.611729    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77" event={"ID":"d085667e-8ce6-402b-9d65-9f3ebf25abbb","Type":"ContainerDied","Data":"b9740b21c868f5646e74ae6a7189b1009622fb2e0733812ef3061e1190fae68c"}
Sep 16 14:02:28.611826 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.611769    2548 scope.go:117] "RemoveContainer" containerID="c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd"
Sep 16 14:02:28.613222 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.613197    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75" event={"ID":"eed57543-ec93-4351-8f3f-5c4d5a33bf98","Type":"ContainerStarted","Data":"3715f25335d92fe6f8c1569409f424c3858c10e4ca1349e95fa8575ffda3a4c1"}
Sep 16 14:02:28.626908 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.626886    2548 scope.go:117] "RemoveContainer" containerID="d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711"
Sep 16 14:02:28.638532 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.636705    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:02:28.638532 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.637434    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:02:28.639075 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.639054    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:02:28.648414 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.648386    2548 scope.go:117] "RemoveContainer" containerID="c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd"
Sep 16 14:02:28.648849 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:02:28.648822    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd\": container with ID starting with c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd not found: ID does not exist" containerID="c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd"
Sep 16 14:02:28.648957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.648853    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd"} err="failed to get container status \"c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd\": rpc error: code = NotFound desc = could not find container \"c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd\": container with ID starting with c4cf09e8e25ffdaa5b8caa9a60ee262406c0d4a78ba3a0d282634b7707cc92cd not found: ID does not exist"
Sep 16 14:02:28.648957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.648869    2548 scope.go:117] "RemoveContainer" containerID="d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711"
Sep 16 14:02:28.649186 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:02:28.649166    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711\": container with ID starting with d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711 not found: ID does not exist" containerID="d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711"
Sep 16 14:02:28.649260 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.649193    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711"} err="failed to get container status \"d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711\": rpc error: code = NotFound desc = could not find container \"d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711\": container with ID starting with d8c38ea18d90ab8cc7cb3e4a1e8aaf198db92ba13d6c99810e63f6f482e6a711 not found: ID does not exist"
Sep 16 14:02:28.680597 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.680533    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"]
Sep 16 14:02:28.684104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:28.684079    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b52a89-openshift-default-c9c56d7b8-gnh77"]
Sep 16 14:02:29.621553 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:29.621521    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75" event={"ID":"eed57543-ec93-4351-8f3f-5c4d5a33bf98","Type":"ContainerStarted","Data":"9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7"}
Sep 16 14:02:29.669419 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:29.669369    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75" podStartSLOduration=2.669356139 podStartE2EDuration="2.669356139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:29.668829858 +0000 UTC m=+1172.296311233" watchObservedRunningTime="2026-09-16 14:02:29.669356139 +0000 UTC m=+1172.296837514"
Sep 16 14:02:29.969994 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:29.969915    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" path="/var/lib/kubelet/pods/d085667e-8ce6-402b-9d65-9f3ebf25abbb/volumes"
Sep 16 14:02:33.506992 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:33.506962    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"]
Sep 16 14:02:33.513515 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:33.513490    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-95743e-29826120-65g4c"]
Sep 16 14:02:33.950623 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:33.950592    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"]
Sep 16 14:02:33.953723 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:33.953080    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-95743e-2lk99"]
Sep 16 14:02:33.983535 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:33.980890    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf44835-5514-4aee-b989-9626d54318c3" path="/var/lib/kubelet/pods/0cf44835-5514-4aee-b989-9626d54318c3/volumes"
Sep 16 14:02:33.983535 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:33.981401    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec" path="/var/lib/kubelet/pods/2ffdf8fc-0843-494f-8b41-7b84e5e7d6ec/volumes"
Sep 16 14:02:38.320011 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:38.319978    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:38.320412 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:38.320327    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:38.333078 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:38.333057    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:38.684268 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:38.684220    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:02:42.201349 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:42.201310    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"]
Sep 16 14:02:42.201732 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:42.201533    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy" containerID="cri-o://6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8" gracePeriod=30
Sep 16 14:02:43.129754 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.129723    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"]
Sep 16 14:02:43.130347 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.130331    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy"
Sep 16 14:02:43.130403 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.130350    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d085667e-8ce6-402b-9d65-9f3ebf25abbb" containerName="istio-proxy"
Sep 16 14:02:43.133662 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.133640    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.136323 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.136304    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-46224d-openshift-default-dockercfg-r9xm8\""
Sep 16 14:02:43.149424 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.149400    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"]
Sep 16 14:02:43.193789 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.193761    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-podinfo\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.193918 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.193797    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-envoy\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.193918 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.193835    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-certs\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.193918 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.193858    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz82q\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-kube-api-access-lz82q\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.193918 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.193888    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-data\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.193918 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.193911    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-socket\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.194204 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.193929    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istiod-ca-cert\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.194204 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.194001    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-credential-socket\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.194204 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.194057    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-token\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.294571 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294536    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-socket\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294578    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istiod-ca-cert\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294613    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-credential-socket\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294657    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-token\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294735    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-podinfo\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294767    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-envoy\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294827    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-certs\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294857    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz82q\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-kube-api-access-lz82q\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.294890    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-data\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295475 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.295077    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-credential-socket\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295475 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.295109    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-socket\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295475 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.295329    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-certs\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295475 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.295389    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-data\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.295475 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.295440    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istiod-ca-cert\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.297140 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.297114    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-envoy\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.297443 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.297425    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-podinfo\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.304659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.304636    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-token\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.304912 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.304885    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz82q\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-kube-api-access-lz82q\") pod \"e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") " pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.443742 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.443656    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:43.728908 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.728878    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"]
Sep 16 14:02:43.730312 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:02:43.730286    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f11ea68_63f3_47bd_9d30_d16f3fe3fefe.slice/crio-472b51c3626ddea807f46d19ccc987b7eeaa753c01a9c611249df20339f73f43 WatchSource:0}: Error finding container 472b51c3626ddea807f46d19ccc987b7eeaa753c01a9c611249df20339f73f43: Status 404 returned error can't find the container with id 472b51c3626ddea807f46d19ccc987b7eeaa753c01a9c611249df20339f73f43
Sep 16 14:02:43.732373 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.732337    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:43.732469 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.732398    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:43.732469 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:43.732426    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:44.709978 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:44.709944    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" event={"ID":"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe","Type":"ContainerStarted","Data":"c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345"}
Sep 16 14:02:44.710413 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:44.709998    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" event={"ID":"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe","Type":"ContainerStarted","Data":"472b51c3626ddea807f46d19ccc987b7eeaa753c01a9c611249df20339f73f43"}
Sep 16 14:02:45.444702 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:45.444662    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:45.449727 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:45.449703    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:45.476637 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:45.476585    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podStartSLOduration=2.4765672690000002 podStartE2EDuration="2.476567269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:44.750746765 +0000 UTC m=+1187.378228139" watchObservedRunningTime="2026-09-16 14:02:45.476567269 +0000 UTC m=+1188.104048644"
Sep 16 14:02:45.715808 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:45.715726    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:45.716581 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:45.716565    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:02:47.458319 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.458288    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 14:02:47.528800 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.528724    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-envoy\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.528944 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.528801    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw7qg\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-kube-api-access-mw7qg\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.528944 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.528896    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-certs\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.528944 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.528929    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-podinfo\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.529108 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.528983    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-credential-socket\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.529108 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529014    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istiod-ca-cert\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.529108 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529052    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-data\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.529108 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529097    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-socket\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.529338 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529122    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-token\") pod \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\" (UID: \"8c97b100-dcfc-47e3-9df5-d720d2b3ac96\") "
Sep 16 14:02:47.529338 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529201    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:47.529544 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529508    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.529672 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529561    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:02:47.529900 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529865    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:47.530009 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.529964    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:47.530142 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.530075    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:47.531864 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.531841    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:47.531992 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.531969    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-kube-api-access-mw7qg" (OuterVolumeSpecName: "kube-api-access-mw7qg") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "kube-api-access-mw7qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:47.532330 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.532306    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:47.532399 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.532380    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c97b100-dcfc-47e3-9df5-d720d2b3ac96" (UID: "8c97b100-dcfc-47e3-9df5-d720d2b3ac96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:02:47.630316 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.630284    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.630316 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.630312    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.630316 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.630323    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.630522 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.630331    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.630522 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.630341    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.630522 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.630350    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.630522 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.630359    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.630522 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.630372    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7qg\" (UniqueName: \"kubernetes.io/projected/8c97b100-dcfc-47e3-9df5-d720d2b3ac96-kube-api-access-mw7qg\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:02:47.728830 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.728793    2548 generic.go:356] "Generic (PLEG): container finished" podID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerID="6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8" exitCode=0
Sep 16 14:02:47.729018 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.728846    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"
Sep 16 14:02:47.729018 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.728875    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" event={"ID":"8c97b100-dcfc-47e3-9df5-d720d2b3ac96","Type":"ContainerDied","Data":"6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8"}
Sep 16 14:02:47.729018 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.728944    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr" event={"ID":"8c97b100-dcfc-47e3-9df5-d720d2b3ac96","Type":"ContainerDied","Data":"aeaa93c7227143740a4d9a10b9e377610b19a0c2bc835167d137d9df7ccbb1a2"}
Sep 16 14:02:47.729018 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.728976    2548 scope.go:117] "RemoveContainer" containerID="6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8"
Sep 16 14:02:47.742712 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.742695    2548 scope.go:117] "RemoveContainer" containerID="0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61"
Sep 16 14:02:47.751159 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.751142    2548 scope.go:117] "RemoveContainer" containerID="6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8"
Sep 16 14:02:47.751464 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:02:47.751446    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8\": container with ID starting with 6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8 not found: ID does not exist" containerID="6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8"
Sep 16 14:02:47.751542 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.751472    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8"} err="failed to get container status \"6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8\": rpc error: code = NotFound desc = could not find container \"6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8\": container with ID starting with 6e81091372cdaf77a592f15fdbe8357b578d57b89306b989a4adf8dd1345fdc8 not found: ID does not exist"
Sep 16 14:02:47.751542 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.751488    2548 scope.go:117] "RemoveContainer" containerID="0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61"
Sep 16 14:02:47.751743 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:02:47.751725    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61\": container with ID starting with 0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61 not found: ID does not exist" containerID="0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61"
Sep 16 14:02:47.751802 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.751753    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61"} err="failed to get container status \"0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61\": rpc error: code = NotFound desc = could not find container \"0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61\": container with ID starting with 0029d5d97696ebca157a22c7b43b04b0327f39b5a54117b3d0071c620a2c6f61 not found: ID does not exist"
Sep 16 14:02:47.756193 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.756174    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"]
Sep 16 14:02:47.761922 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.761902    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-95743e-openshift-default-5d46f95db-nckpr"]
Sep 16 14:02:47.971075 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:47.971044    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" path="/var/lib/kubelet/pods/8c97b100-dcfc-47e3-9df5-d720d2b3ac96/volumes"
Sep 16 14:02:52.295267 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.293158    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"]
Sep 16 14:02:52.295267 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.293798    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy"
Sep 16 14:02:52.295267 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.293817    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy"
Sep 16 14:02:52.295267 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.293833    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy"
Sep 16 14:02:52.295267 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.293842    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy"
Sep 16 14:02:52.295267 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.293961    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy"
Sep 16 14:02:52.295267 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.293979    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c97b100-dcfc-47e3-9df5-d720d2b3ac96" containerName="istio-proxy"
Sep 16 14:02:52.303912 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.303890    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:52.308878 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.308852    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-46224d-dockercfg-fl7lc\""
Sep 16 14:02:52.309054 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.308906    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-46224d\""
Sep 16 14:02:52.315880 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.315859    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"]
Sep 16 14:02:52.329075 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.329047    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"]
Sep 16 14:02:52.333232 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.332937    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:52.347072 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.347052    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"]
Sep 16 14:02:52.374838 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.374814    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n\" (UID: \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:52.374985 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.374854    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25m5\" (UniqueName: \"kubernetes.io/projected/64e0f993-3400-4864-a3f5-898c734236f7-kube-api-access-k25m5\") pod \"payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd\" (UID: \"64e0f993-3400-4864-a3f5-898c734236f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:52.374985 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.374923    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e0f993-3400-4864-a3f5-898c734236f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd\" (UID: \"64e0f993-3400-4864-a3f5-898c734236f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:52.375058 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.375006    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8cn2\" (UniqueName: \"kubernetes.io/projected/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-kube-api-access-q8cn2\") pod \"payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n\" (UID: \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:52.475755 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.475722    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n\" (UID: \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:52.475932 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.475777    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25m5\" (UniqueName: \"kubernetes.io/projected/64e0f993-3400-4864-a3f5-898c734236f7-kube-api-access-k25m5\") pod \"payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd\" (UID: \"64e0f993-3400-4864-a3f5-898c734236f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:52.475932 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.475833    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e0f993-3400-4864-a3f5-898c734236f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd\" (UID: \"64e0f993-3400-4864-a3f5-898c734236f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:52.475932 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.475924    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8cn2\" (UniqueName: \"kubernetes.io/projected/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-kube-api-access-q8cn2\") pod \"payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n\" (UID: \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:52.476670 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.476632    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n\" (UID: \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:52.476810 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.476710    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e0f993-3400-4864-a3f5-898c734236f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd\" (UID: \"64e0f993-3400-4864-a3f5-898c734236f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:52.495282 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.494034    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25m5\" (UniqueName: \"kubernetes.io/projected/64e0f993-3400-4864-a3f5-898c734236f7-kube-api-access-k25m5\") pod \"payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd\" (UID: \"64e0f993-3400-4864-a3f5-898c734236f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:52.499276 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.495809    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8cn2\" (UniqueName: \"kubernetes.io/projected/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-kube-api-access-q8cn2\") pod \"payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n\" (UID: \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:52.618650 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.618615    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:52.643186 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.643160    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:52.911389 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.911364    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"]
Sep 16 14:02:52.913370 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:02:52.913344    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa6b6b8_51d3_4d64_94e6_e65cb1681e49.slice/crio-1855b82bb9edd5a9255ba4b070eeb6d962351f05caefe9a31973c410c046de49 WatchSource:0}: Error finding container 1855b82bb9edd5a9255ba4b070eeb6d962351f05caefe9a31973c410c046de49: Status 404 returned error can't find the container with id 1855b82bb9edd5a9255ba4b070eeb6d962351f05caefe9a31973c410c046de49
Sep 16 14:02:52.960502 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:52.960478    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"]
Sep 16 14:02:52.961337 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:02:52.961312    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e0f993_3400_4864_a3f5_898c734236f7.slice/crio-9e7ddbaaf140a31f3aec1eb803da5925add2c7ca09873a41035533b12c47c0a8 WatchSource:0}: Error finding container 9e7ddbaaf140a31f3aec1eb803da5925add2c7ca09873a41035533b12c47c0a8: Status 404 returned error can't find the container with id 9e7ddbaaf140a31f3aec1eb803da5925add2c7ca09873a41035533b12c47c0a8
Sep 16 14:02:53.772051 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:53.771960    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd" event={"ID":"64e0f993-3400-4864-a3f5-898c734236f7","Type":"ContainerStarted","Data":"87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71"}
Sep 16 14:02:53.772482 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:53.772127    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:02:53.772482 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:53.772146    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd" event={"ID":"64e0f993-3400-4864-a3f5-898c734236f7","Type":"ContainerStarted","Data":"9e7ddbaaf140a31f3aec1eb803da5925add2c7ca09873a41035533b12c47c0a8"}
Sep 16 14:02:53.773384 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:53.773363    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n" event={"ID":"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49","Type":"ContainerStarted","Data":"ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1"}
Sep 16 14:02:53.773457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:53.773399    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n" event={"ID":"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49","Type":"ContainerStarted","Data":"1855b82bb9edd5a9255ba4b070eeb6d962351f05caefe9a31973c410c046de49"}
Sep 16 14:02:53.773529 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:53.773510    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:02:53.796074 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:53.796014    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd" podStartSLOduration=1.7959943900000002 podStartE2EDuration="1.79599439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:53.79283945 +0000 UTC m=+1196.420320825" watchObservedRunningTime="2026-09-16 14:02:53.79599439 +0000 UTC m=+1196.423475764"
Sep 16 14:02:53.817600 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:53.816549    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n" podStartSLOduration=1.816526235 podStartE2EDuration="1.816526235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:53.814973969 +0000 UTC m=+1196.442455342" watchObservedRunningTime="2026-09-16 14:02:53.816526235 +0000 UTC m=+1196.444007607"
Sep 16 14:02:57.870216 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:57.870185    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 14:02:57.871994 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:57.871969    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 14:02:58.489391 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:58.489348    2548 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:39688->10.133.0.59:15021: read: connection reset by peer" start-of-body=
Sep 16 14:02:58.489578 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:58.489443    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:39688->10.133.0.59:15021: read: connection reset by peer"
Sep 16 14:02:58.808637 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:58.808542    2548 generic.go:356] "Generic (PLEG): container finished" podID="0117d88d-499d-4196-badf-6e77bd473b17" containerID="1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0" exitCode=0
Sep 16 14:02:58.808637 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:58.808615    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" event={"ID":"0117d88d-499d-4196-badf-6e77bd473b17","Type":"ContainerDied","Data":"1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0"}
Sep 16 14:02:58.809040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:58.809025    2548 scope.go:117] "RemoveContainer" containerID="1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0"
Sep 16 14:02:58.810279 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:58.810236    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:58.810375 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:58.810317    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:58.810375 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:58.810348    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:02:59.816755 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:02:59.816723    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" event={"ID":"0117d88d-499d-4196-badf-6e77bd473b17","Type":"ContainerStarted","Data":"3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12"}
Sep 16 14:03:00.327635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:00.327593    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:03:00.327824 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:00.327787    2548 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 16 14:03:00.327879 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:00.327848    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 16 14:03:01.327938 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:01.327903    2548 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 16 14:03:01.328333 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:01.327961    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 16 14:03:02.327588 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:02.327551    2548 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 16 14:03:02.327763 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:02.327611    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 16 14:03:03.327420 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:03.327385    2548 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 16 14:03:03.327781 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:03.327442    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 16 14:03:04.327977 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:04.327933    2548 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 16 14:03:04.328398 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:04.328000    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 16 14:03:04.783750 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:04.783712    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:03:04.783930 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:04.783785    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:03:05.331346 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:05.331316    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:03:05.331703 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:05.331562    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:03:05.332275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:05.332241    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:03:15.986793 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:15.986761    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"]
Sep 16 14:03:15.989386 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:15.989370    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:15.992908 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:15.992887    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae7c3682\""
Sep 16 14:03:16.003008 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.002982    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"]
Sep 16 14:03:16.071213 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.071187    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4rm\" (UniqueName: \"kubernetes.io/projected/908aa4c9-328b-4cf5-885c-82cfe2ea7926-kube-api-access-mp4rm\") pod \"payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf\" (UID: \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\") " pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:16.071371 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.071262    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908aa4c9-328b-4cf5-885c-82cfe2ea7926-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf\" (UID: \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\") " pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:16.171974 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.171935    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908aa4c9-328b-4cf5-885c-82cfe2ea7926-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf\" (UID: \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\") " pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:16.172140 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.172062    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4rm\" (UniqueName: \"kubernetes.io/projected/908aa4c9-328b-4cf5-885c-82cfe2ea7926-kube-api-access-mp4rm\") pod \"payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf\" (UID: \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\") " pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:16.172768 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.172748    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908aa4c9-328b-4cf5-885c-82cfe2ea7926-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf\" (UID: \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\") " pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:16.183372 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.183344    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4rm\" (UniqueName: \"kubernetes.io/projected/908aa4c9-328b-4cf5-885c-82cfe2ea7926-kube-api-access-mp4rm\") pod \"payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf\" (UID: \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\") " pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:16.300803 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.300729    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:16.573631 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.573607    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"]
Sep 16 14:03:16.575065 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:03:16.575026    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908aa4c9_328b_4cf5_885c_82cfe2ea7926.slice/crio-57861a9ab2188ed7fdb2443ead5da031f2410ef5efa90e2840cab42e95f898a2 WatchSource:0}: Error finding container 57861a9ab2188ed7fdb2443ead5da031f2410ef5efa90e2840cab42e95f898a2: Status 404 returned error can't find the container with id 57861a9ab2188ed7fdb2443ead5da031f2410ef5efa90e2840cab42e95f898a2
Sep 16 14:03:16.740282 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.740204    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 14:03:16.740456 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.740279    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 14:03:16.930023 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.929958    2548 generic.go:356] "Generic (PLEG): container finished" podID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerID="c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345" exitCode=0
Sep 16 14:03:16.930161 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.930037    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" event={"ID":"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe","Type":"ContainerDied","Data":"c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345"}
Sep 16 14:03:16.930602 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.930583    2548 scope.go:117] "RemoveContainer" containerID="c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345"
Sep 16 14:03:16.931818 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.931787    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf" event={"ID":"908aa4c9-328b-4cf5-885c-82cfe2ea7926","Type":"ContainerStarted","Data":"13adc20a145374e6cea3d03af39ca45c324e37e68b252b046b297786da5c5be3"}
Sep 16 14:03:16.931911 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.931843    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf" event={"ID":"908aa4c9-328b-4cf5-885c-82cfe2ea7926","Type":"ContainerStarted","Data":"57861a9ab2188ed7fdb2443ead5da031f2410ef5efa90e2840cab42e95f898a2"}
Sep 16 14:03:16.931911 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.931895    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:16.931911 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.931889    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:03:16.932029 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.931952    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:03:16.932029 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.931982    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:03:16.988843 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:16.988801    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf" podStartSLOduration=1.988785643 podStartE2EDuration="1.988785643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:16.987159306 +0000 UTC m=+1219.614640682" watchObservedRunningTime="2026-09-16 14:03:16.988785643 +0000 UTC m=+1219.616267016"
Sep 16 14:03:17.940149 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:17.940115    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" event={"ID":"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe","Type":"ContainerStarted","Data":"87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8"}
Sep 16 14:03:18.444349 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:18.444309    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:03:18.444724 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:18.444499    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 14:03:18.444724 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:18.444543    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 14:03:19.445072 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:19.445038    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 14:03:19.445494 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:19.445101    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 14:03:20.444503 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:20.444468    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 14:03:20.444719 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:20.444532    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 14:03:21.444454 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:21.444416    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 14:03:21.444839 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:21.444471    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 14:03:22.444606 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:22.444569    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 14:03:22.444996 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:22.444621    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 14:03:23.444199 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:23.444160    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 16 14:03:23.444396 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:23.444219    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 16 14:03:24.448187 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.448157    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:03:24.448658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.448550    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:03:24.449290 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.449272    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:03:24.846913 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.846833    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"]
Sep 16 14:03:24.849485 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.849468    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:24.852691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.852665    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b4884e\""
Sep 16 14:03:24.862374 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.862351    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"]
Sep 16 14:03:24.964161 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.964123    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdxg\" (UniqueName: \"kubernetes.io/projected/85246908-84cf-4090-be06-728822a42556-kube-api-access-gtdxg\") pod \"payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7\" (UID: \"85246908-84cf-4090-be06-728822a42556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:24.964439 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:24.964417    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85246908-84cf-4090-be06-728822a42556-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7\" (UID: \"85246908-84cf-4090-be06-728822a42556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:25.066034 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.066001    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdxg\" (UniqueName: \"kubernetes.io/projected/85246908-84cf-4090-be06-728822a42556-kube-api-access-gtdxg\") pod \"payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7\" (UID: \"85246908-84cf-4090-be06-728822a42556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:25.066034 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.066039    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85246908-84cf-4090-be06-728822a42556-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7\" (UID: \"85246908-84cf-4090-be06-728822a42556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:25.067106 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.067080    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85246908-84cf-4090-be06-728822a42556-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7\" (UID: \"85246908-84cf-4090-be06-728822a42556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:25.079760 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.079738    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdxg\" (UniqueName: \"kubernetes.io/projected/85246908-84cf-4090-be06-728822a42556-kube-api-access-gtdxg\") pod \"payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7\" (UID: \"85246908-84cf-4090-be06-728822a42556\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:25.161099 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.161077    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:25.435002 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.434969    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"]
Sep 16 14:03:25.435328 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:03:25.435305    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85246908_84cf_4090_be06_728822a42556.slice/crio-fad6d228e03a956e3d90bb8755b91fe203e2b8cd81ceb5280215305af18ee169 WatchSource:0}: Error finding container fad6d228e03a956e3d90bb8755b91fe203e2b8cd81ceb5280215305af18ee169: Status 404 returned error can't find the container with id fad6d228e03a956e3d90bb8755b91fe203e2b8cd81ceb5280215305af18ee169
Sep 16 14:03:25.991385 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.991346    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7" event={"ID":"85246908-84cf-4090-be06-728822a42556","Type":"ContainerStarted","Data":"c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73"}
Sep 16 14:03:25.991800 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.991541    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:25.991800 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:25.991589    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7" event={"ID":"85246908-84cf-4090-be06-728822a42556","Type":"ContainerStarted","Data":"fad6d228e03a956e3d90bb8755b91fe203e2b8cd81ceb5280215305af18ee169"}
Sep 16 14:03:26.011425 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:26.011378    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7" podStartSLOduration=2.011361644 podStartE2EDuration="2.011361644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:26.010709398 +0000 UTC m=+1228.638190774" watchObservedRunningTime="2026-09-16 14:03:26.011361644 +0000 UTC m=+1228.638843021"
Sep 16 14:03:27.942269 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:27.942212    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:03:37.000985 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:37.000952    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:03:46.136630 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.136600    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"]
Sep 16 14:03:46.139436 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.139415    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.142106 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.142083    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ae7c3682\"/\"openshift-service-ca.crt\""
Sep 16 14:03:46.143146 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.143127    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ae7c3682\"/\"kube-root-ca.crt\""
Sep 16 14:03:46.143270 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.143132    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ae7c3682\"/\"e2e-lifecycle-model-ae7c3682-kserve-self-signed-certs\""
Sep 16 14:03:46.143270 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.143207    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ae7c3682\"/\"default-dockercfg-8m6cf\""
Sep 16 14:03:46.150370 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.150349    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"]
Sep 16 14:03:46.181599 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.181574    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-home\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.181697 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.181641    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-dshm\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.181697 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.181684    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-model-cache\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.181810 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.181772    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhzt\" (UniqueName: \"kubernetes.io/projected/37578be1-90ae-4cd6-b4df-3099a6430f93-kube-api-access-2fhzt\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.181875 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.181830    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-tmp-dir\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.181875 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.181860    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37578be1-90ae-4cd6-b4df-3099a6430f93-tls-certs\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.283637 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.283604    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhzt\" (UniqueName: \"kubernetes.io/projected/37578be1-90ae-4cd6-b4df-3099a6430f93-kube-api-access-2fhzt\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.283795 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.283654    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-tmp-dir\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.283795 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.283676    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37578be1-90ae-4cd6-b4df-3099a6430f93-tls-certs\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.283795 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.283771    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-home\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.283957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.283826    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-dshm\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.283957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.283867    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-model-cache\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.284154 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.284131    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-tmp-dir\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.284235 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.284160    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-home\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.284235 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.284189    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-model-cache\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.286063 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.286036    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-dshm\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.286316 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.286299    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37578be1-90ae-4cd6-b4df-3099a6430f93-tls-certs\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.294945 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.294918    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhzt\" (UniqueName: \"kubernetes.io/projected/37578be1-90ae-4cd6-b4df-3099a6430f93-kube-api-access-2fhzt\") pod \"e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") " pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.449650 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.449573    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:46.726001 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:46.725976    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"]
Sep 16 14:03:46.727712 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:03:46.727685    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37578be1_90ae_4cd6_b4df_3099a6430f93.slice/crio-5141da8c09ad821f0797643205be1c15cdcb6cb099f638032cfe8b8cf9ae6fda WatchSource:0}: Error finding container 5141da8c09ad821f0797643205be1c15cdcb6cb099f638032cfe8b8cf9ae6fda: Status 404 returned error can't find the container with id 5141da8c09ad821f0797643205be1c15cdcb6cb099f638032cfe8b8cf9ae6fda
Sep 16 14:03:47.109861 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:47.109773    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs" event={"ID":"37578be1-90ae-4cd6-b4df-3099a6430f93","Type":"ContainerStarted","Data":"62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574"}
Sep 16 14:03:47.109861 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:47.109823    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs" event={"ID":"37578be1-90ae-4cd6-b4df-3099a6430f93","Type":"ContainerStarted","Data":"5141da8c09ad821f0797643205be1c15cdcb6cb099f638032cfe8b8cf9ae6fda"}
Sep 16 14:03:47.136976 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:47.136923    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs" podStartSLOduration=1.136908702 podStartE2EDuration="1.136908702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:47.135344157 +0000 UTC m=+1249.762825531" watchObservedRunningTime="2026-09-16 14:03:47.136908702 +0000 UTC m=+1249.764390109"
Sep 16 14:03:56.044892 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.044805    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"]
Sep 16 14:03:56.051267 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.049806    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.054367 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.054342    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"kube-root-ca.crt\""
Sep 16 14:03:56.054779 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.054758    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"auth-test-model-46224d-kserve-self-signed-certs\""
Sep 16 14:03:56.055037 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.054785    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"default-dockercfg-7g6qm\""
Sep 16 14:03:56.055330 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.055313    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"openshift-service-ca.crt\""
Sep 16 14:03:56.057943 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.057921    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"]
Sep 16 14:03:56.172607 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.172575    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-model-cache\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.172780 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.172621    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-dshm\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.172780 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.172640    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tls-certs\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.172780 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.172734    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tmp-dir\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.172904 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.172793    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-home\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.172904 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.172818    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db5z7\" (UniqueName: \"kubernetes.io/projected/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-kube-api-access-db5z7\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273400 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273359    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-model-cache\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273567 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273517    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-dshm\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273567 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273551    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tls-certs\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273686 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273590    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tmp-dir\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273686 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273634    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-home\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273686 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273652    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db5z7\" (UniqueName: \"kubernetes.io/projected/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-kube-api-access-db5z7\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273839 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273735    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-model-cache\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273965 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273941    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tmp-dir\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.273965 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.273959    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-home\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.275749 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.275730    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-dshm\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.275947 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.275930    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tls-certs\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.284547 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.284524    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db5z7\" (UniqueName: \"kubernetes.io/projected/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-kube-api-access-db5z7\") pod \"auth-test-model-46224d-kserve-564fc9ddc-jfwf7\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") " pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.365840 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.365812    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:03:56.450643 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.450611    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:56.450789 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.450670    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:56.464431 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.464410    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:56.668485 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:56.668460    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"]
Sep 16 14:03:56.669553 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:03:56.669524    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a4d11db_0ff6_43f0_ac17_d65b31976ba0.slice/crio-3b64b44e2c0bd5ca50a983a5b625fe5360c2bdc2217119d805653a65d690f653 WatchSource:0}: Error finding container 3b64b44e2c0bd5ca50a983a5b625fe5360c2bdc2217119d805653a65d690f653: Status 404 returned error can't find the container with id 3b64b44e2c0bd5ca50a983a5b625fe5360c2bdc2217119d805653a65d690f653
Sep 16 14:03:57.163840 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:57.163791    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7" event={"ID":"8a4d11db-0ff6-43f0-ac17-d65b31976ba0","Type":"ContainerStarted","Data":"58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1"}
Sep 16 14:03:57.164326 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:57.163852    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7" event={"ID":"8a4d11db-0ff6-43f0-ac17-d65b31976ba0","Type":"ContainerStarted","Data":"3b64b44e2c0bd5ca50a983a5b625fe5360c2bdc2217119d805653a65d690f653"}
Sep 16 14:03:57.175580 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:57.175557    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:03:57.185195 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:57.185152    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7" podStartSLOduration=1.185137852 podStartE2EDuration="1.185137852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:57.184438176 +0000 UTC m=+1259.811919551" watchObservedRunningTime="2026-09-16 14:03:57.185137852 +0000 UTC m=+1259.812619225"
Sep 16 14:03:57.947526 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:57.947495    2548 scope.go:117] "RemoveContainer" containerID="c1335faa61c11c6dcf17b79612ce66e250ae5c2945603763871b66a7ca096bb4"
Sep 16 14:03:57.956319 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:03:57.956296    2548 scope.go:117] "RemoveContainer" containerID="1ef68645c67b0a1fe9e7c2c994781b92227384c2a2d3264100e7ba1c77124c8c"
Sep 16 14:04:06.366301 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:06.366232    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:04:06.366714 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:06.366314    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:04:06.378973 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:06.378949    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:04:07.228985 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:07.228953    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:04:45.654450 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.654413    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"]
Sep 16 14:04:45.657812 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.657796    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"
Sep 16 14:04:45.660435 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.660412    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:04:45.660566 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.660415    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:04:45.668257 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.668223    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"]
Sep 16 14:04:45.747601 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.747569    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2ce451-3f34-4df4-b5ab-7970c75f71a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5\" (UID: \"0f2ce451-3f34-4df4-b5ab-7970c75f71a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"
Sep 16 14:04:45.849138 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.849107    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2ce451-3f34-4df4-b5ab-7970c75f71a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5\" (UID: \"0f2ce451-3f34-4df4-b5ab-7970c75f71a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"
Sep 16 14:04:45.849695 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.849675    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2ce451-3f34-4df4-b5ab-7970c75f71a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5\" (UID: \"0f2ce451-3f34-4df4-b5ab-7970c75f71a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"
Sep 16 14:04:45.969509 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:45.969427    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"
Sep 16 14:04:46.247893 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:46.247868    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"]
Sep 16 14:04:46.250402 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:04:46.250353    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2ce451_3f34_4df4_b5ab_7970c75f71a3.slice/crio-9d5f394c3f076201623b7308908a1d1e19a9b2bb08fec15a9a63a610990a06ca WatchSource:0}: Error finding container 9d5f394c3f076201623b7308908a1d1e19a9b2bb08fec15a9a63a610990a06ca: Status 404 returned error can't find the container with id 9d5f394c3f076201623b7308908a1d1e19a9b2bb08fec15a9a63a610990a06ca
Sep 16 14:04:46.449786 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:46.449749    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5" event={"ID":"0f2ce451-3f34-4df4-b5ab-7970c75f71a3","Type":"ContainerStarted","Data":"9d5f394c3f076201623b7308908a1d1e19a9b2bb08fec15a9a63a610990a06ca"}
Sep 16 14:04:47.456911 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:47.456878    2548 generic.go:356] "Generic (PLEG): container finished" podID="0f2ce451-3f34-4df4-b5ab-7970c75f71a3" containerID="277bc0cda300e0ced46d37a16203331b1d92a2c446c6bd7cc79d8ff3a793dc96" exitCode=0
Sep 16 14:04:47.457272 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:47.456941    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5" event={"ID":"0f2ce451-3f34-4df4-b5ab-7970c75f71a3","Type":"ContainerDied","Data":"277bc0cda300e0ced46d37a16203331b1d92a2c446c6bd7cc79d8ff3a793dc96"}
Sep 16 14:04:48.595960 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:48.595936    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"
Sep 16 14:04:48.673903 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:48.673876    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2ce451-3f34-4df4-b5ab-7970c75f71a3-maas-api-service-ca\") pod \"0f2ce451-3f34-4df4-b5ab-7970c75f71a3\" (UID: \"0f2ce451-3f34-4df4-b5ab-7970c75f71a3\") "
Sep 16 14:04:48.674224 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:48.674201    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2ce451-3f34-4df4-b5ab-7970c75f71a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f2ce451-3f34-4df4-b5ab-7970c75f71a3" (UID: "0f2ce451-3f34-4df4-b5ab-7970c75f71a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:04:48.775189 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:48.775105    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2ce451-3f34-4df4-b5ab-7970c75f71a3-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:04:49.469788 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:49.469762    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"
Sep 16 14:04:49.469955 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:49.469792    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5" event={"ID":"0f2ce451-3f34-4df4-b5ab-7970c75f71a3","Type":"ContainerDied","Data":"9d5f394c3f076201623b7308908a1d1e19a9b2bb08fec15a9a63a610990a06ca"}
Sep 16 14:04:49.469955 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:49.469831    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d5f394c3f076201623b7308908a1d1e19a9b2bb08fec15a9a63a610990a06ca"
Sep 16 14:04:57.980452 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:04:57.980418    2548 scope.go:117] "RemoveContainer" containerID="9363a296cbd0fe6c3e5f1c59a8b29be29d4b1c1e2a380fbd7c143350ebc601bd"
Sep 16 14:05:01.383978 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.383927    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"]
Sep 16 14:05:01.384546 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.384166    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf" podUID="908aa4c9-328b-4cf5-885c-82cfe2ea7926" containerName="payload-processing" containerID="cri-o://13adc20a145374e6cea3d03af39ca45c324e37e68b252b046b297786da5c5be3" gracePeriod=30
Sep 16 14:05:01.412448 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:05:01.412409    2548 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908aa4c9_328b_4cf5_885c_82cfe2ea7926.slice/crio-conmon-13adc20a145374e6cea3d03af39ca45c324e37e68b252b046b297786da5c5be3.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 14:05:01.538903 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.538869    2548 generic.go:356] "Generic (PLEG): container finished" podID="908aa4c9-328b-4cf5-885c-82cfe2ea7926" containerID="13adc20a145374e6cea3d03af39ca45c324e37e68b252b046b297786da5c5be3" exitCode=0
Sep 16 14:05:01.539096 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.538943    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf" event={"ID":"908aa4c9-328b-4cf5-885c-82cfe2ea7926","Type":"ContainerDied","Data":"13adc20a145374e6cea3d03af39ca45c324e37e68b252b046b297786da5c5be3"}
Sep 16 14:05:01.643194 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.643141    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:05:01.688757 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.688717    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"]
Sep 16 14:05:01.693444 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.693421    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae7c3682-7xdr5"]
Sep 16 14:05:01.800887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.800857    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908aa4c9-328b-4cf5-885c-82cfe2ea7926-plugins-config-volume\") pod \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\" (UID: \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\") "
Sep 16 14:05:01.801066 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.800913    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp4rm\" (UniqueName: \"kubernetes.io/projected/908aa4c9-328b-4cf5-885c-82cfe2ea7926-kube-api-access-mp4rm\") pod \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\" (UID: \"908aa4c9-328b-4cf5-885c-82cfe2ea7926\") "
Sep 16 14:05:01.801334 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.801302    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908aa4c9-328b-4cf5-885c-82cfe2ea7926-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "908aa4c9-328b-4cf5-885c-82cfe2ea7926" (UID: "908aa4c9-328b-4cf5-885c-82cfe2ea7926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:05:01.803096 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.803063    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908aa4c9-328b-4cf5-885c-82cfe2ea7926-kube-api-access-mp4rm" (OuterVolumeSpecName: "kube-api-access-mp4rm") pod "908aa4c9-328b-4cf5-885c-82cfe2ea7926" (UID: "908aa4c9-328b-4cf5-885c-82cfe2ea7926"). InnerVolumeSpecName "kube-api-access-mp4rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:01.902588 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.902508    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908aa4c9-328b-4cf5-885c-82cfe2ea7926-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:05:01.902588 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.902536    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4rm\" (UniqueName: \"kubernetes.io/projected/908aa4c9-328b-4cf5-885c-82cfe2ea7926-kube-api-access-mp4rm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:05:01.970962 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:01.970920    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2ce451-3f34-4df4-b5ab-7970c75f71a3" path="/var/lib/kubelet/pods/0f2ce451-3f34-4df4-b5ab-7970c75f71a3/volumes"
Sep 16 14:05:02.546016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:02.545982    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf" event={"ID":"908aa4c9-328b-4cf5-885c-82cfe2ea7926","Type":"ContainerDied","Data":"57861a9ab2188ed7fdb2443ead5da031f2410ef5efa90e2840cab42e95f898a2"}
Sep 16 14:05:02.546410 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:02.546020    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"
Sep 16 14:05:02.546410 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:02.546041    2548 scope.go:117] "RemoveContainer" containerID="13adc20a145374e6cea3d03af39ca45c324e37e68b252b046b297786da5c5be3"
Sep 16 14:05:02.571000 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:02.570975    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"]
Sep 16 14:05:02.574278 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:02.574255    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae7c3682-5fccb5f5c-4pxqf"]
Sep 16 14:05:03.971264 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:03.971223    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908aa4c9-328b-4cf5-885c-82cfe2ea7926" path="/var/lib/kubelet/pods/908aa4c9-328b-4cf5-885c-82cfe2ea7926/volumes"
Sep 16 14:05:09.828919 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:09.828880    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"]
Sep 16 14:05:19.376648 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.376615    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"]
Sep 16 14:05:19.377234 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.377214    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2ce451-3f34-4df4-b5ab-7970c75f71a3" containerName="revoke-keys"
Sep 16 14:05:19.377368 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.377236    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2ce451-3f34-4df4-b5ab-7970c75f71a3" containerName="revoke-keys"
Sep 16 14:05:19.377368 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.377336    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908aa4c9-328b-4cf5-885c-82cfe2ea7926" containerName="payload-processing"
Sep 16 14:05:19.377368 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.377348    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="908aa4c9-328b-4cf5-885c-82cfe2ea7926" containerName="payload-processing"
Sep 16 14:05:19.377513 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.377468    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2ce451-3f34-4df4-b5ab-7970c75f71a3" containerName="revoke-keys"
Sep 16 14:05:19.377513 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.377487    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="908aa4c9-328b-4cf5-885c-82cfe2ea7926" containerName="payload-processing"
Sep 16 14:05:19.382143 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.382121    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.385024 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.384999    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035c64\"/\"embedding-w0-035c64-kserve-self-signed-certs\""
Sep 16 14:05:19.402091 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.402059    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"]
Sep 16 14:05:19.482885 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.481142    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbc30f9-e376-4b54-bb39-6f0c62303486-tls-certs\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.482885 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.481226    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-tmp-dir\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.482885 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.481372    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-model-cache\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.482885 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.481400    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9hmk\" (UniqueName: \"kubernetes.io/projected/fcbc30f9-e376-4b54-bb39-6f0c62303486-kube-api-access-w9hmk\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.482885 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.481473    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-home\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.482885 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.481523    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-dshm\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.582934 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.582898    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbc30f9-e376-4b54-bb39-6f0c62303486-tls-certs\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.583122 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.582946    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-tmp-dir\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.583122 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.583024    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-model-cache\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.583122 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.583047    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9hmk\" (UniqueName: \"kubernetes.io/projected/fcbc30f9-e376-4b54-bb39-6f0c62303486-kube-api-access-w9hmk\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.583122 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.583094    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-home\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.583122 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.583119    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-dshm\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.583573 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.583548    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-tmp-dir\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.583697 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.583621    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-model-cache\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.583697 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.583668    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-home\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.585410 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.585389    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-dshm\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.585706 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.585686    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbc30f9-e376-4b54-bb39-6f0c62303486-tls-certs\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.610166 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.610139    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9hmk\" (UniqueName: \"kubernetes.io/projected/fcbc30f9-e376-4b54-bb39-6f0c62303486-kube-api-access-w9hmk\") pod \"embedding-w0-035c64-kserve-6b87d44769-g8kcd\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") " pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.692217 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.692150    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:19.978861 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:19.978834    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"]
Sep 16 14:05:19.980468 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:05:19.980437    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcbc30f9_e376_4b54_bb39_6f0c62303486.slice/crio-53f477ec4f2bf5e1226cf5c8a012656ce124b45886d441e096604e6fbd5c3078 WatchSource:0}: Error finding container 53f477ec4f2bf5e1226cf5c8a012656ce124b45886d441e096604e6fbd5c3078: Status 404 returned error can't find the container with id 53f477ec4f2bf5e1226cf5c8a012656ce124b45886d441e096604e6fbd5c3078
Sep 16 14:05:20.649913 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:20.649870    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd" event={"ID":"fcbc30f9-e376-4b54-bb39-6f0c62303486","Type":"ContainerStarted","Data":"edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30"}
Sep 16 14:05:20.650371 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:20.649919    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd" event={"ID":"fcbc30f9-e376-4b54-bb39-6f0c62303486","Type":"ContainerStarted","Data":"53f477ec4f2bf5e1226cf5c8a012656ce124b45886d441e096604e6fbd5c3078"}
Sep 16 14:05:20.671026 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:20.670981    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd" podStartSLOduration=1.6709663620000001 podStartE2EDuration="1.670966362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:05:20.66833608 +0000 UTC m=+1343.295817458" watchObservedRunningTime="2026-09-16 14:05:20.670966362 +0000 UTC m=+1343.298447753"
Sep 16 14:05:24.847765 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:24.847722    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs" podUID="37578be1-90ae-4cd6-b4df-3099a6430f93" containerName="main" containerID="cri-o://62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574" gracePeriod=45
Sep 16 14:05:25.108123 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.108066    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:05:25.241226 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241198    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-model-cache\") pod \"37578be1-90ae-4cd6-b4df-3099a6430f93\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") "
Sep 16 14:05:25.241439 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241286    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-home\") pod \"37578be1-90ae-4cd6-b4df-3099a6430f93\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") "
Sep 16 14:05:25.241439 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241335    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-dshm\") pod \"37578be1-90ae-4cd6-b4df-3099a6430f93\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") "
Sep 16 14:05:25.241439 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241394    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-tmp-dir\") pod \"37578be1-90ae-4cd6-b4df-3099a6430f93\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") "
Sep 16 14:05:25.241439 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241414    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37578be1-90ae-4cd6-b4df-3099a6430f93-tls-certs\") pod \"37578be1-90ae-4cd6-b4df-3099a6430f93\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") "
Sep 16 14:05:25.241659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241459    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fhzt\" (UniqueName: \"kubernetes.io/projected/37578be1-90ae-4cd6-b4df-3099a6430f93-kube-api-access-2fhzt\") pod \"37578be1-90ae-4cd6-b4df-3099a6430f93\" (UID: \"37578be1-90ae-4cd6-b4df-3099a6430f93\") "
Sep 16 14:05:25.241659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241523    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-model-cache" (OuterVolumeSpecName: "model-cache") pod "37578be1-90ae-4cd6-b4df-3099a6430f93" (UID: "37578be1-90ae-4cd6-b4df-3099a6430f93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:25.241659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241561    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-home" (OuterVolumeSpecName: "home") pod "37578be1-90ae-4cd6-b4df-3099a6430f93" (UID: "37578be1-90ae-4cd6-b4df-3099a6430f93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:25.241659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241632    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37578be1-90ae-4cd6-b4df-3099a6430f93" (UID: "37578be1-90ae-4cd6-b4df-3099a6430f93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:25.241868 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241804    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:05:25.241868 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241816    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:05:25.241868 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.241825    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:05:25.243544 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.243510    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37578be1-90ae-4cd6-b4df-3099a6430f93-kube-api-access-2fhzt" (OuterVolumeSpecName: "kube-api-access-2fhzt") pod "37578be1-90ae-4cd6-b4df-3099a6430f93" (UID: "37578be1-90ae-4cd6-b4df-3099a6430f93"). InnerVolumeSpecName "kube-api-access-2fhzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:25.243650 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.243576    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37578be1-90ae-4cd6-b4df-3099a6430f93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37578be1-90ae-4cd6-b4df-3099a6430f93" (UID: "37578be1-90ae-4cd6-b4df-3099a6430f93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:05:25.243650 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.243581    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-dshm" (OuterVolumeSpecName: "dshm") pod "37578be1-90ae-4cd6-b4df-3099a6430f93" (UID: "37578be1-90ae-4cd6-b4df-3099a6430f93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:25.342869 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.342838    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37578be1-90ae-4cd6-b4df-3099a6430f93-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:05:25.342869 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.342863    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fhzt\" (UniqueName: \"kubernetes.io/projected/37578be1-90ae-4cd6-b4df-3099a6430f93-kube-api-access-2fhzt\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:05:25.342869 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.342873    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37578be1-90ae-4cd6-b4df-3099a6430f93-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:05:25.679303 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.679272    2548 generic.go:356] "Generic (PLEG): container finished" podID="37578be1-90ae-4cd6-b4df-3099a6430f93" containerID="62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574" exitCode=0
Sep 16 14:05:25.679470 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.679312    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs" event={"ID":"37578be1-90ae-4cd6-b4df-3099a6430f93","Type":"ContainerDied","Data":"62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574"}
Sep 16 14:05:25.679470 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.679329    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"
Sep 16 14:05:25.679470 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.679362    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs" event={"ID":"37578be1-90ae-4cd6-b4df-3099a6430f93","Type":"ContainerDied","Data":"5141da8c09ad821f0797643205be1c15cdcb6cb099f638032cfe8b8cf9ae6fda"}
Sep 16 14:05:25.679470 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.679383    2548 scope.go:117] "RemoveContainer" containerID="62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574"
Sep 16 14:05:25.696631 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.696613    2548 scope.go:117] "RemoveContainer" containerID="62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574"
Sep 16 14:05:25.696911 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:05:25.696893    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574\": container with ID starting with 62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574 not found: ID does not exist" containerID="62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574"
Sep 16 14:05:25.696958 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.696920    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574"} err="failed to get container status \"62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574\": rpc error: code = NotFound desc = could not find container \"62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574\": container with ID starting with 62a5c9cf8dcd737d10892852d382bc3007e5839c9264b7512fa8796d0a90c574 not found: ID does not exist"
Sep 16 14:05:25.711047 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.711020    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"]
Sep 16 14:05:25.715311 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.715286    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ae7c3682/e2e-lifecycle-model-ae7c3682-kserve-5b6f6b9f9f-56fcs"]
Sep 16 14:05:25.970483 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:25.970402    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37578be1-90ae-4cd6-b4df-3099a6430f93" path="/var/lib/kubelet/pods/37578be1-90ae-4cd6-b4df-3099a6430f93/volumes"
Sep 16 14:05:29.692318 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:29.692277    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:29.692759 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:29.692331    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:29.705157 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:29.705130    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:05:29.716488 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:05:29.716457    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:06:23.617888 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.617847    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"]
Sep 16 14:06:23.618527 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.618507    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37578be1-90ae-4cd6-b4df-3099a6430f93" containerName="main"
Sep 16 14:06:23.618618 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.618530    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="37578be1-90ae-4cd6-b4df-3099a6430f93" containerName="main"
Sep 16 14:06:23.618672 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.618641    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="37578be1-90ae-4cd6-b4df-3099a6430f93" containerName="main"
Sep 16 14:06:23.623933 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.623910    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"
Sep 16 14:06:23.627811 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.627792    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:06:23.628719 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.628698    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:06:23.639168 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.639149    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"]
Sep 16 14:06:23.710356 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.710315    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c60bf-c433-491c-bc42-a17fed75135f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5\" (UID: \"fb5c60bf-c433-491c-bc42-a17fed75135f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"
Sep 16 14:06:23.811707 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.811672    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c60bf-c433-491c-bc42-a17fed75135f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5\" (UID: \"fb5c60bf-c433-491c-bc42-a17fed75135f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"
Sep 16 14:06:23.812350 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.812329    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c60bf-c433-491c-bc42-a17fed75135f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5\" (UID: \"fb5c60bf-c433-491c-bc42-a17fed75135f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"
Sep 16 14:06:23.934045 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:23.933966    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"
Sep 16 14:06:24.223971 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:24.223905    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"]
Sep 16 14:06:24.226403 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:06:24.226372    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5c60bf_c433_491c_bc42_a17fed75135f.slice/crio-4ee81dc86b3fc074099e241a9c8cce6fda791a641ed7f49cd1156b478f4b8904 WatchSource:0}: Error finding container 4ee81dc86b3fc074099e241a9c8cce6fda791a641ed7f49cd1156b478f4b8904: Status 404 returned error can't find the container with id 4ee81dc86b3fc074099e241a9c8cce6fda791a641ed7f49cd1156b478f4b8904
Sep 16 14:06:25.018008 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:25.017977    2548 generic.go:356] "Generic (PLEG): container finished" podID="fb5c60bf-c433-491c-bc42-a17fed75135f" containerID="4677a6212db916c8fee4a20b3cdddbf0fb2cb19cb29bab13b64abb8bb919a21e" exitCode=0
Sep 16 14:06:25.018400 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:25.018041    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5" event={"ID":"fb5c60bf-c433-491c-bc42-a17fed75135f","Type":"ContainerDied","Data":"4677a6212db916c8fee4a20b3cdddbf0fb2cb19cb29bab13b64abb8bb919a21e"}
Sep 16 14:06:25.018400 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:25.018078    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5" event={"ID":"fb5c60bf-c433-491c-bc42-a17fed75135f","Type":"ContainerStarted","Data":"4ee81dc86b3fc074099e241a9c8cce6fda791a641ed7f49cd1156b478f4b8904"}
Sep 16 14:06:26.163811 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:26.163787    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"
Sep 16 14:06:26.236134 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:26.236016    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c60bf-c433-491c-bc42-a17fed75135f-maas-api-service-ca\") pod \"fb5c60bf-c433-491c-bc42-a17fed75135f\" (UID: \"fb5c60bf-c433-491c-bc42-a17fed75135f\") "
Sep 16 14:06:26.236405 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:26.236380    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5c60bf-c433-491c-bc42-a17fed75135f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb5c60bf-c433-491c-bc42-a17fed75135f" (UID: "fb5c60bf-c433-491c-bc42-a17fed75135f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:06:26.236552 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:26.236538    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5c60bf-c433-491c-bc42-a17fed75135f-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:27.030367 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:27.030324    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5" event={"ID":"fb5c60bf-c433-491c-bc42-a17fed75135f","Type":"ContainerDied","Data":"4ee81dc86b3fc074099e241a9c8cce6fda791a641ed7f49cd1156b478f4b8904"}
Sep 16 14:06:27.030541 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:27.030377    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"
Sep 16 14:06:27.030541 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:27.030384    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ee81dc86b3fc074099e241a9c8cce6fda791a641ed7f49cd1156b478f4b8904"
Sep 16 14:06:28.522686 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:28.522654    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"]
Sep 16 14:06:28.541893 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:28.541864    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"]
Sep 16 14:06:39.422720 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.422685    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"]
Sep 16 14:06:39.423127 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.422900    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg" podUID="a8d82ddf-4e94-4aad-a48e-05703d94231c" containerName="payload-processing" containerID="cri-o://ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0" gracePeriod=30
Sep 16 14:06:39.696205 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.696176    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 14:06:39.768743 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.768710    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"]
Sep 16 14:06:39.772157 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.772131    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-035c64-stzd5"]
Sep 16 14:06:39.867579 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.867544    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d82ddf-4e94-4aad-a48e-05703d94231c-plugins-config-volume\") pod \"a8d82ddf-4e94-4aad-a48e-05703d94231c\" (UID: \"a8d82ddf-4e94-4aad-a48e-05703d94231c\") "
Sep 16 14:06:39.867762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.867639    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjjh8\" (UniqueName: \"kubernetes.io/projected/a8d82ddf-4e94-4aad-a48e-05703d94231c-kube-api-access-zjjh8\") pod \"a8d82ddf-4e94-4aad-a48e-05703d94231c\" (UID: \"a8d82ddf-4e94-4aad-a48e-05703d94231c\") "
Sep 16 14:06:39.867992 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.867966    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d82ddf-4e94-4aad-a48e-05703d94231c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8d82ddf-4e94-4aad-a48e-05703d94231c" (UID: "a8d82ddf-4e94-4aad-a48e-05703d94231c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:06:39.869798 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.869774    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d82ddf-4e94-4aad-a48e-05703d94231c-kube-api-access-zjjh8" (OuterVolumeSpecName: "kube-api-access-zjjh8") pod "a8d82ddf-4e94-4aad-a48e-05703d94231c" (UID: "a8d82ddf-4e94-4aad-a48e-05703d94231c"). InnerVolumeSpecName "kube-api-access-zjjh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:06:39.968594 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.968519    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d82ddf-4e94-4aad-a48e-05703d94231c-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:39.968594 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.968549    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjjh8\" (UniqueName: \"kubernetes.io/projected/a8d82ddf-4e94-4aad-a48e-05703d94231c-kube-api-access-zjjh8\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:39.970315 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:39.970293    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5c60bf-c433-491c-bc42-a17fed75135f" path="/var/lib/kubelet/pods/fb5c60bf-c433-491c-bc42-a17fed75135f/volumes"
Sep 16 14:06:40.115444 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.115406    2548 generic.go:356] "Generic (PLEG): container finished" podID="a8d82ddf-4e94-4aad-a48e-05703d94231c" containerID="ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0" exitCode=0
Sep 16 14:06:40.115598 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.115453    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"
Sep 16 14:06:40.115598 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.115492    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg" event={"ID":"a8d82ddf-4e94-4aad-a48e-05703d94231c","Type":"ContainerDied","Data":"ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0"}
Sep 16 14:06:40.115598 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.115550    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg" event={"ID":"a8d82ddf-4e94-4aad-a48e-05703d94231c","Type":"ContainerDied","Data":"99354fc3b0e19cb74206d8b7c6af51338cc1d3805005fa95cb8a94ba01441476"}
Sep 16 14:06:40.115598 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.115575    2548 scope.go:117] "RemoveContainer" containerID="ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0"
Sep 16 14:06:40.124682 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.124665    2548 scope.go:117] "RemoveContainer" containerID="ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0"
Sep 16 14:06:40.124962 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:06:40.124942    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0\": container with ID starting with ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0 not found: ID does not exist" containerID="ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0"
Sep 16 14:06:40.125013 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.124970    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0"} err="failed to get container status \"ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0\": rpc error: code = NotFound desc = could not find container \"ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0\": container with ID starting with ea4d3ce13b488e242e4317f2cae95cf2a87bcd9c97383bd50b48f686b5e23fa0 not found: ID does not exist"
Sep 16 14:06:40.151850 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.151817    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"]
Sep 16 14:06:40.167959 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:40.167936    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035c64-54f8b8b6b9-sxtdg"]
Sep 16 14:06:41.971168 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:41.971135    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d82ddf-4e94-4aad-a48e-05703d94231c" path="/var/lib/kubelet/pods/a8d82ddf-4e94-4aad-a48e-05703d94231c/volumes"
Sep 16 14:06:43.542879 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:43.542841    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75" podUID="eed57543-ec93-4351-8f3f-5c4d5a33bf98" containerName="main" containerID="cri-o://9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7" gracePeriod=45
Sep 16 14:06:43.562634 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:43.562605    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd" podUID="fcbc30f9-e376-4b54-bb39-6f0c62303486" containerName="main" containerID="cri-o://edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30" gracePeriod=45
Sep 16 14:06:43.844830 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:43.844806    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:06:43.847632 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:43.847612    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:06:44.008449 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008420    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-dshm\") pod \"fcbc30f9-e376-4b54-bb39-6f0c62303486\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") "
Sep 16 14:06:44.008449 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008451    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-tmp-dir\") pod \"fcbc30f9-e376-4b54-bb39-6f0c62303486\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") "
Sep 16 14:06:44.008691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008479    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-dshm\") pod \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") "
Sep 16 14:06:44.008691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008506    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbc30f9-e376-4b54-bb39-6f0c62303486-tls-certs\") pod \"fcbc30f9-e376-4b54-bb39-6f0c62303486\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") "
Sep 16 14:06:44.008691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008543    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-home\") pod \"fcbc30f9-e376-4b54-bb39-6f0c62303486\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") "
Sep 16 14:06:44.008691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008575    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-home\") pod \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") "
Sep 16 14:06:44.008691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008665    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrjrr\" (UniqueName: \"kubernetes.io/projected/eed57543-ec93-4351-8f3f-5c4d5a33bf98-kube-api-access-rrjrr\") pod \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") "
Sep 16 14:06:44.008691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008690    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tmp-dir\") pod \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") "
Sep 16 14:06:44.009011 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008706    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcbc30f9-e376-4b54-bb39-6f0c62303486" (UID: "fcbc30f9-e376-4b54-bb39-6f0c62303486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:44.009011 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008747    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9hmk\" (UniqueName: \"kubernetes.io/projected/fcbc30f9-e376-4b54-bb39-6f0c62303486-kube-api-access-w9hmk\") pod \"fcbc30f9-e376-4b54-bb39-6f0c62303486\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") "
Sep 16 14:06:44.009011 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008802    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tls-certs\") pod \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") "
Sep 16 14:06:44.009011 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008829    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-model-cache\") pod \"fcbc30f9-e376-4b54-bb39-6f0c62303486\" (UID: \"fcbc30f9-e376-4b54-bb39-6f0c62303486\") "
Sep 16 14:06:44.009011 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.008858    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-model-cache\") pod \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\" (UID: \"eed57543-ec93-4351-8f3f-5c4d5a33bf98\") "
Sep 16 14:06:44.009290 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.009216    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eed57543-ec93-4351-8f3f-5c4d5a33bf98" (UID: "eed57543-ec93-4351-8f3f-5c4d5a33bf98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:44.009359 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.009297    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.009519 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.009485    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-home" (OuterVolumeSpecName: "home") pod "fcbc30f9-e376-4b54-bb39-6f0c62303486" (UID: "fcbc30f9-e376-4b54-bb39-6f0c62303486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:44.009757 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.009734    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-model-cache" (OuterVolumeSpecName: "model-cache") pod "eed57543-ec93-4351-8f3f-5c4d5a33bf98" (UID: "eed57543-ec93-4351-8f3f-5c4d5a33bf98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:44.009956 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.009933    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-home" (OuterVolumeSpecName: "home") pod "eed57543-ec93-4351-8f3f-5c4d5a33bf98" (UID: "eed57543-ec93-4351-8f3f-5c4d5a33bf98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:44.010522 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.010497    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcbc30f9-e376-4b54-bb39-6f0c62303486" (UID: "fcbc30f9-e376-4b54-bb39-6f0c62303486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:44.011116 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.011079    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-dshm" (OuterVolumeSpecName: "dshm") pod "eed57543-ec93-4351-8f3f-5c4d5a33bf98" (UID: "eed57543-ec93-4351-8f3f-5c4d5a33bf98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:44.011305 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.011242    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed57543-ec93-4351-8f3f-5c4d5a33bf98-kube-api-access-rrjrr" (OuterVolumeSpecName: "kube-api-access-rrjrr") pod "eed57543-ec93-4351-8f3f-5c4d5a33bf98" (UID: "eed57543-ec93-4351-8f3f-5c4d5a33bf98"). InnerVolumeSpecName "kube-api-access-rrjrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:06:44.011426 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.011406    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbc30f9-e376-4b54-bb39-6f0c62303486-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcbc30f9-e376-4b54-bb39-6f0c62303486" (UID: "fcbc30f9-e376-4b54-bb39-6f0c62303486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:06:44.011600 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.011569    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eed57543-ec93-4351-8f3f-5c4d5a33bf98" (UID: "eed57543-ec93-4351-8f3f-5c4d5a33bf98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:06:44.011720 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.011697    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbc30f9-e376-4b54-bb39-6f0c62303486-kube-api-access-w9hmk" (OuterVolumeSpecName: "kube-api-access-w9hmk") pod "fcbc30f9-e376-4b54-bb39-6f0c62303486" (UID: "fcbc30f9-e376-4b54-bb39-6f0c62303486"). InnerVolumeSpecName "kube-api-access-w9hmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:06:44.012001 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.011982    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-dshm" (OuterVolumeSpecName: "dshm") pod "fcbc30f9-e376-4b54-bb39-6f0c62303486" (UID: "fcbc30f9-e376-4b54-bb39-6f0c62303486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110526    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9hmk\" (UniqueName: \"kubernetes.io/projected/fcbc30f9-e376-4b54-bb39-6f0c62303486-kube-api-access-w9hmk\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110554    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110565    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110573    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110580    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110587    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110594    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbc30f9-e376-4b54-bb39-6f0c62303486-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110601    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbc30f9-e376-4b54-bb39-6f0c62303486-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110607    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110991 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110615    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrjrr\" (UniqueName: \"kubernetes.io/projected/eed57543-ec93-4351-8f3f-5c4d5a33bf98-kube-api-access-rrjrr\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.110991 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.110632    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eed57543-ec93-4351-8f3f-5c4d5a33bf98-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:06:44.148787 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.148749    2548 generic.go:356] "Generic (PLEG): container finished" podID="eed57543-ec93-4351-8f3f-5c4d5a33bf98" containerID="9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7" exitCode=0
Sep 16 14:06:44.148947 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.148792    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"
Sep 16 14:06:44.148947 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.148831    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75" event={"ID":"eed57543-ec93-4351-8f3f-5c4d5a33bf98","Type":"ContainerDied","Data":"9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7"}
Sep 16 14:06:44.148947 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.148881    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75" event={"ID":"eed57543-ec93-4351-8f3f-5c4d5a33bf98","Type":"ContainerDied","Data":"3715f25335d92fe6f8c1569409f424c3858c10e4ca1349e95fa8575ffda3a4c1"}
Sep 16 14:06:44.148947 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.148900    2548 scope.go:117] "RemoveContainer" containerID="9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7"
Sep 16 14:06:44.150261 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.150231    2548 generic.go:356] "Generic (PLEG): container finished" podID="fcbc30f9-e376-4b54-bb39-6f0c62303486" containerID="edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30" exitCode=0
Sep 16 14:06:44.150358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.150280    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd" event={"ID":"fcbc30f9-e376-4b54-bb39-6f0c62303486","Type":"ContainerDied","Data":"edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30"}
Sep 16 14:06:44.150358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.150330    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd" event={"ID":"fcbc30f9-e376-4b54-bb39-6f0c62303486","Type":"ContainerDied","Data":"53f477ec4f2bf5e1226cf5c8a012656ce124b45886d441e096604e6fbd5c3078"}
Sep 16 14:06:44.150358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.150291    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"
Sep 16 14:06:44.159548 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.159533    2548 scope.go:117] "RemoveContainer" containerID="9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7"
Sep 16 14:06:44.159802 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:06:44.159782    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7\": container with ID starting with 9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7 not found: ID does not exist" containerID="9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7"
Sep 16 14:06:44.159903 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.159807    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7"} err="failed to get container status \"9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7\": rpc error: code = NotFound desc = could not find container \"9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7\": container with ID starting with 9ab762dc414483295b9838d44ec76b8220094c5dcab61af9009a57153279e1a7 not found: ID does not exist"
Sep 16 14:06:44.159903 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.159823    2548 scope.go:117] "RemoveContainer" containerID="edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30"
Sep 16 14:06:44.167505 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.167490    2548 scope.go:117] "RemoveContainer" containerID="edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30"
Sep 16 14:06:44.167735 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:06:44.167722    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30\": container with ID starting with edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30 not found: ID does not exist" containerID="edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30"
Sep 16 14:06:44.167775 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.167740    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30"} err="failed to get container status \"edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30\": rpc error: code = NotFound desc = could not find container \"edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30\": container with ID starting with edd442cf9a789ddf129f608b16c40211fa2c30b71e3c652832d59a9c4fe1ad30 not found: ID does not exist"
Sep 16 14:06:44.174920 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.174899    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"]
Sep 16 14:06:44.180497 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.180478    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/distinct-2-w0-035c64-kserve-5f67db767-p8n75"]
Sep 16 14:06:44.200200 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.200175    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"]
Sep 16 14:06:44.206226 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:44.206205    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035c64/embedding-w0-035c64-kserve-6b87d44769-g8kcd"]
Sep 16 14:06:45.977228 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:45.977188    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed57543-ec93-4351-8f3f-5c4d5a33bf98" path="/var/lib/kubelet/pods/eed57543-ec93-4351-8f3f-5c4d5a33bf98/volumes"
Sep 16 14:06:45.977783 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:45.977762    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcbc30f9-e376-4b54-bb39-6f0c62303486" path="/var/lib/kubelet/pods/fcbc30f9-e376-4b54-bb39-6f0c62303486/volumes"
Sep 16 14:06:58.016461 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:58.016379    2548 scope.go:117] "RemoveContainer" containerID="ef1c4642b361d70b380a6dcd2fbf770affa7abf9f18ad9a54138ff4a9e26a641"
Sep 16 14:06:58.027226 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:06:58.027206    2548 scope.go:117] "RemoveContainer" containerID="95ae036a2b46dd6eaf42654b0fe61c39fbc1ebb76658d4413f015d01cb69d4d1"
Sep 16 14:07:01.844587 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.844557    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"]
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.844950    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed57543-ec93-4351-8f3f-5c4d5a33bf98" containerName="main"
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.844961    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed57543-ec93-4351-8f3f-5c4d5a33bf98" containerName="main"
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.844978    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcbc30f9-e376-4b54-bb39-6f0c62303486" containerName="main"
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.844984    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcbc30f9-e376-4b54-bb39-6f0c62303486" containerName="main"
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.844992    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5c60bf-c433-491c-bc42-a17fed75135f" containerName="revoke-keys"
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.844998    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5c60bf-c433-491c-bc42-a17fed75135f" containerName="revoke-keys"
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.845014    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d82ddf-4e94-4aad-a48e-05703d94231c" containerName="payload-processing"
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.845019    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d82ddf-4e94-4aad-a48e-05703d94231c" containerName="payload-processing"
Sep 16 14:07:01.845077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.845077    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed57543-ec93-4351-8f3f-5c4d5a33bf98" containerName="main"
Sep 16 14:07:01.845549 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.845085    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d82ddf-4e94-4aad-a48e-05703d94231c" containerName="payload-processing"
Sep 16 14:07:01.845549 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.845095    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5c60bf-c433-491c-bc42-a17fed75135f" containerName="revoke-keys"
Sep 16 14:07:01.845549 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.845106    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcbc30f9-e376-4b54-bb39-6f0c62303486" containerName="main"
Sep 16 14:07:01.848170 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.848153    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.856885 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.856864    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7f75a3-openshift-default-dockercfg-7lshj\""
Sep 16 14:07:01.862969 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.862950    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-data\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.863043 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.862990    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-envoy\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.863111 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.863044    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6btv\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-kube-api-access-s6btv\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.863111 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.863090    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-podinfo\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.863223 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.863111    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-certs\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.863223 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.863132    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-socket\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.863223 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.863182    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istiod-ca-cert\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.863398 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.863234    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-token\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.863398 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.863272    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-credential-socket\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.895181 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.895148    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"]
Sep 16 14:07:01.964233 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964196    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-socket\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964448 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964270    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istiod-ca-cert\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964448 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964343    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-token\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964448 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964384    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-credential-socket\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964448 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964431    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-data\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964675 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964650    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-socket\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964732 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964667    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-envoy\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964794 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964731    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6btv\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-kube-api-access-s6btv\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964794 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964740    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-credential-socket\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964794 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964765    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-data\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964952 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964811    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-podinfo\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.964952 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.964843    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-certs\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.965165 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.965145    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-certs\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.965551 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.965529    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istiod-ca-cert\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.967400 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.967353    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-envoy\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.967851 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.967813    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-podinfo\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.987229 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.987197    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-token\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:01.987389 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:01.987369    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6btv\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-kube-api-access-s6btv\") pod \"e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") " pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:02.159112 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:02.159078    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:02.460476 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:02.460398    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"]
Sep 16 14:07:02.464014 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:07:02.463987    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1a1cda_8d33_40ac_b1ed_7a29f01aec7c.slice/crio-7bcb253fa633adea35e8af358b10cf43c81a73571a428131fbfb685b036bfdc4 WatchSource:0}: Error finding container 7bcb253fa633adea35e8af358b10cf43c81a73571a428131fbfb685b036bfdc4: Status 404 returned error can't find the container with id 7bcb253fa633adea35e8af358b10cf43c81a73571a428131fbfb685b036bfdc4
Sep 16 14:07:02.466172 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:02.466140    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:02.466284 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:02.466206    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:02.466284 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:02.466235    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:03.261732 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:03.261696    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" event={"ID":"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c","Type":"ContainerStarted","Data":"ba3d18c136c1141ad1e31bfd3c0c422527e292f62d7a75c46a7d1e9ffe8545a9"}
Sep 16 14:07:03.262103 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:03.261752    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" event={"ID":"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c","Type":"ContainerStarted","Data":"7bcb253fa633adea35e8af358b10cf43c81a73571a428131fbfb685b036bfdc4"}
Sep 16 14:07:03.293409 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:03.293363    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" podStartSLOduration=2.293348066 podStartE2EDuration="2.293348066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:03.2902631 +0000 UTC m=+1445.917744472" watchObservedRunningTime="2026-09-16 14:07:03.293348066 +0000 UTC m=+1445.920829440"
Sep 16 14:07:04.159858 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:04.159817    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:04.165073 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:04.165048    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:04.267104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:04.267075    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:04.268110 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:04.268091    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:19.510615 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.510580    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"]
Sep 16 14:07:19.513964 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.513946    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"
Sep 16 14:07:19.518727 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.518700    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:07:19.518845 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.518729    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:07:19.525915 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.525895    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"]
Sep 16 14:07:19.636445 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.636415    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247c2435-613e-41b5-8f6b-6b80f071081e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j\" (UID: \"247c2435-613e-41b5-8f6b-6b80f071081e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"
Sep 16 14:07:19.736983 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.736950    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247c2435-613e-41b5-8f6b-6b80f071081e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j\" (UID: \"247c2435-613e-41b5-8f6b-6b80f071081e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"
Sep 16 14:07:19.737608 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.737587    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247c2435-613e-41b5-8f6b-6b80f071081e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j\" (UID: \"247c2435-613e-41b5-8f6b-6b80f071081e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"
Sep 16 14:07:19.824944 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:19.824862    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"
Sep 16 14:07:20.118616 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:20.118585    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"]
Sep 16 14:07:20.360852 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:20.360816    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j" event={"ID":"247c2435-613e-41b5-8f6b-6b80f071081e","Type":"ContainerStarted","Data":"95a33effd0bfbdffbc6f5d3b326c6a680f9cda5c40202370624cd33217112051"}
Sep 16 14:07:21.368612 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:21.368582    2548 generic.go:356] "Generic (PLEG): container finished" podID="247c2435-613e-41b5-8f6b-6b80f071081e" containerID="47806588c33fc3ebf8ba3acdf1b76952a4445a70bed77321ed5f5e3ae8ba7f69" exitCode=0
Sep 16 14:07:21.368994 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:21.368667    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j" event={"ID":"247c2435-613e-41b5-8f6b-6b80f071081e","Type":"ContainerDied","Data":"47806588c33fc3ebf8ba3acdf1b76952a4445a70bed77321ed5f5e3ae8ba7f69"}
Sep 16 14:07:22.551725 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:22.551696    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"
Sep 16 14:07:22.672121 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:22.672045    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247c2435-613e-41b5-8f6b-6b80f071081e-maas-api-service-ca\") pod \"247c2435-613e-41b5-8f6b-6b80f071081e\" (UID: \"247c2435-613e-41b5-8f6b-6b80f071081e\") "
Sep 16 14:07:22.672410 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:22.672387    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247c2435-613e-41b5-8f6b-6b80f071081e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "247c2435-613e-41b5-8f6b-6b80f071081e" (UID: "247c2435-613e-41b5-8f6b-6b80f071081e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:07:22.772835 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:22.772800    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247c2435-613e-41b5-8f6b-6b80f071081e-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:23.383348 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:23.383312    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j" event={"ID":"247c2435-613e-41b5-8f6b-6b80f071081e","Type":"ContainerDied","Data":"95a33effd0bfbdffbc6f5d3b326c6a680f9cda5c40202370624cd33217112051"}
Sep 16 14:07:23.383526 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:23.383376    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a33effd0bfbdffbc6f5d3b326c6a680f9cda5c40202370624cd33217112051"
Sep 16 14:07:23.383526 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:23.383438    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"
Sep 16 14:07:30.436837 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:30.436800    2548 generic.go:356] "Generic (PLEG): container finished" podID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerID="ba3d18c136c1141ad1e31bfd3c0c422527e292f62d7a75c46a7d1e9ffe8545a9" exitCode=0
Sep 16 14:07:30.437327 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:30.436877    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" event={"ID":"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c","Type":"ContainerDied","Data":"ba3d18c136c1141ad1e31bfd3c0c422527e292f62d7a75c46a7d1e9ffe8545a9"}
Sep 16 14:07:30.437395 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:30.437386    2548 scope.go:117] "RemoveContainer" containerID="ba3d18c136c1141ad1e31bfd3c0c422527e292f62d7a75c46a7d1e9ffe8545a9"
Sep 16 14:07:30.438638 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:30.438606    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:30.438706 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:30.438672    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:30.438742 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:30.438706    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:31.036667 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.036630    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"]
Sep 16 14:07:31.037291 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.037238    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247c2435-613e-41b5-8f6b-6b80f071081e" containerName="revoke-keys"
Sep 16 14:07:31.037291 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.037275    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="247c2435-613e-41b5-8f6b-6b80f071081e" containerName="revoke-keys"
Sep 16 14:07:31.037441 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.037345    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="247c2435-613e-41b5-8f6b-6b80f071081e" containerName="revoke-keys"
Sep 16 14:07:31.039342 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.039325    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.048209 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.048193    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8a4447-openshift-default-dockercfg-kcd2t\""
Sep 16 14:07:31.105098 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.103113    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"]
Sep 16 14:07:31.166943 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.166914    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-envoy\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.167121 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.166958    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83cf247a-4735-4479-8cd6-4f6a6938e306-istiod-ca-cert\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.167121 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.167042    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-podinfo\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.167121 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.167086    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-data\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.167121 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.167102    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttw2\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-kube-api-access-vttw2\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.167371 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.167162    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-token\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.167371 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.167282    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-certs\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.167371 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.167315    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-credential-socket\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.167371 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.167341    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-socket\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.268611 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268576    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83cf247a-4735-4479-8cd6-4f6a6938e306-istiod-ca-cert\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.268790 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268639    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-podinfo\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.268790 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268673    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-data\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.268790 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268710    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttw2\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-kube-api-access-vttw2\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.268790 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268748    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-token\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268823    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-certs\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268847    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-credential-socket\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268876    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-socket\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.268966    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-envoy\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269213 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.269060    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-data\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269304 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.269209    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-credential-socket\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269304 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.269283    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-socket\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269432 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.269411    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83cf247a-4735-4479-8cd6-4f6a6938e306-istiod-ca-cert\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.269496 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.269414    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-certs\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.271070 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.271046    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-envoy\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.271288 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.271268    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-podinfo\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.282642 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.282621    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-token\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.295757 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.295678    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttw2\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-kube-api-access-vttw2\") pod \"e2e-api-b-8a4447-openshift-default-648cdb974-nzg66\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") " pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.350479 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.350448    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:31.467346 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.467305    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" event={"ID":"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c","Type":"ContainerStarted","Data":"a7744bff4c1c795fe6c56b7f1d668f63f6327a30b3779a83b12f6c2e64f25278"}
Sep 16 14:07:31.641875 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.641840    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"]
Sep 16 14:07:31.642032 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:07:31.641983    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cf247a_4735_4479_8cd6_4f6a6938e306.slice/crio-0c6620f14165dff72c6d6449f6064f74733aef9c1f8b4a1e4441bf4eef0b26d8 WatchSource:0}: Error finding container 0c6620f14165dff72c6d6449f6064f74733aef9c1f8b4a1e4441bf4eef0b26d8: Status 404 returned error can't find the container with id 0c6620f14165dff72c6d6449f6064f74733aef9c1f8b4a1e4441bf4eef0b26d8
Sep 16 14:07:31.644742 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.644706    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:31.644814 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.644779    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:31.644814 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:31.644807    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:07:32.159918 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:32.159883    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:32.161296 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:32.161233    2548 patch_prober.go:28] interesting pod/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:07:32.161417 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:32.161305    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 16 14:07:32.474969 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:32.474875    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" event={"ID":"83cf247a-4735-4479-8cd6-4f6a6938e306","Type":"ContainerStarted","Data":"271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf"}
Sep 16 14:07:32.474969 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:32.474931    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" event={"ID":"83cf247a-4735-4479-8cd6-4f6a6938e306","Type":"ContainerStarted","Data":"0c6620f14165dff72c6d6449f6064f74733aef9c1f8b4a1e4441bf4eef0b26d8"}
Sep 16 14:07:32.503527 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:32.503472    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" podStartSLOduration=1.503455531 podStartE2EDuration="1.503455531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:32.500912139 +0000 UTC m=+1475.128393514" watchObservedRunningTime="2026-09-16 14:07:32.503455531 +0000 UTC m=+1475.130936962"
Sep 16 14:07:33.160167 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:33.160134    2548 patch_prober.go:28] interesting pod/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:07:33.160360 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:33.160199    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 16 14:07:33.351306 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:33.351267    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:33.356304 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:33.356278    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:33.484608 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:33.484524    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:33.491400 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:33.491377    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:07:34.159939 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:34.159901    2548 patch_prober.go:28] interesting pod/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:07:34.160116 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:34.159965    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 16 14:07:34.267309 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:34.267262    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:35.160283 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:35.160234    2548 patch_prober.go:28] interesting pod/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:07:35.160724 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:35.160310    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 16 14:07:36.160104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:36.160069    2548 patch_prober.go:28] interesting pod/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 16 14:07:36.160303 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:36.160130    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 16 14:07:37.164202 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:37.164168    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:37.165271 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:37.165229    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:07:45.672420 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:45.672383    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"]
Sep 16 14:07:45.672897 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:45.672568    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw" podUID="87ba6caa-592d-4511-91b3-74bfea7a75ad" containerName="payload-processing" containerID="cri-o://e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab" gracePeriod=30
Sep 16 14:07:45.694479 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:45.694448    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"]
Sep 16 14:07:45.694867 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:45.694833    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff" podUID="9d5b971f-e015-4bb7-9de6-71ef9381b294" containerName="payload-pre-processing" containerID="cri-o://e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6" gracePeriod=30
Sep 16 14:07:46.003325 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.003296    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 14:07:46.006502 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.006481    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 14:07:46.126165 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.126132    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/9d5b971f-e015-4bb7-9de6-71ef9381b294-kube-api-access-s6nhr\") pod \"9d5b971f-e015-4bb7-9de6-71ef9381b294\" (UID: \"9d5b971f-e015-4bb7-9de6-71ef9381b294\") "
Sep 16 14:07:46.126374 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.126178    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b971f-e015-4bb7-9de6-71ef9381b294-plugins-config-volume\") pod \"9d5b971f-e015-4bb7-9de6-71ef9381b294\" (UID: \"9d5b971f-e015-4bb7-9de6-71ef9381b294\") "
Sep 16 14:07:46.126374 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.126195    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba6caa-592d-4511-91b3-74bfea7a75ad-plugins-config-volume\") pod \"87ba6caa-592d-4511-91b3-74bfea7a75ad\" (UID: \"87ba6caa-592d-4511-91b3-74bfea7a75ad\") "
Sep 16 14:07:46.126374 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.126210    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zpzf\" (UniqueName: \"kubernetes.io/projected/87ba6caa-592d-4511-91b3-74bfea7a75ad-kube-api-access-6zpzf\") pod \"87ba6caa-592d-4511-91b3-74bfea7a75ad\" (UID: \"87ba6caa-592d-4511-91b3-74bfea7a75ad\") "
Sep 16 14:07:46.126637 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.126613    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ba6caa-592d-4511-91b3-74bfea7a75ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87ba6caa-592d-4511-91b3-74bfea7a75ad" (UID: "87ba6caa-592d-4511-91b3-74bfea7a75ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:07:46.126733 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.126613    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5b971f-e015-4bb7-9de6-71ef9381b294-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d5b971f-e015-4bb7-9de6-71ef9381b294" (UID: "9d5b971f-e015-4bb7-9de6-71ef9381b294"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:07:46.126733 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.126705    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5b971f-e015-4bb7-9de6-71ef9381b294-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:46.126733 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.126722    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ba6caa-592d-4511-91b3-74bfea7a75ad-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:46.128851 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.128828    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5b971f-e015-4bb7-9de6-71ef9381b294-kube-api-access-s6nhr" (OuterVolumeSpecName: "kube-api-access-s6nhr") pod "9d5b971f-e015-4bb7-9de6-71ef9381b294" (UID: "9d5b971f-e015-4bb7-9de6-71ef9381b294"). InnerVolumeSpecName "kube-api-access-s6nhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:07:46.128959 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.128858    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ba6caa-592d-4511-91b3-74bfea7a75ad-kube-api-access-6zpzf" (OuterVolumeSpecName: "kube-api-access-6zpzf") pod "87ba6caa-592d-4511-91b3-74bfea7a75ad" (UID: "87ba6caa-592d-4511-91b3-74bfea7a75ad"). InnerVolumeSpecName "kube-api-access-6zpzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:07:46.189464 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.189437    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"]
Sep 16 14:07:46.193639 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.193617    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-19203d-75z7j"]
Sep 16 14:07:46.228199 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.228156    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/9d5b971f-e015-4bb7-9de6-71ef9381b294-kube-api-access-s6nhr\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:46.228199 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.228191    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zpzf\" (UniqueName: \"kubernetes.io/projected/87ba6caa-592d-4511-91b3-74bfea7a75ad-kube-api-access-6zpzf\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:46.568923 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.568880    2548 generic.go:356] "Generic (PLEG): container finished" podID="9d5b971f-e015-4bb7-9de6-71ef9381b294" containerID="e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6" exitCode=0
Sep 16 14:07:46.569131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.568942    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"
Sep 16 14:07:46.569131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.568967    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff" event={"ID":"9d5b971f-e015-4bb7-9de6-71ef9381b294","Type":"ContainerDied","Data":"e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6"}
Sep 16 14:07:46.569131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.569070    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff" event={"ID":"9d5b971f-e015-4bb7-9de6-71ef9381b294","Type":"ContainerDied","Data":"edbb52a4c9224b409b2d29e9ebd9adb8793884bc3bcd2923062cb8a61a1d4b1c"}
Sep 16 14:07:46.569131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.569103    2548 scope.go:117] "RemoveContainer" containerID="e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6"
Sep 16 14:07:46.570577 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.570559    2548 generic.go:356] "Generic (PLEG): container finished" podID="87ba6caa-592d-4511-91b3-74bfea7a75ad" containerID="e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab" exitCode=0
Sep 16 14:07:46.570672 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.570598    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw" event={"ID":"87ba6caa-592d-4511-91b3-74bfea7a75ad","Type":"ContainerDied","Data":"e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab"}
Sep 16 14:07:46.570672 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.570631    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw" event={"ID":"87ba6caa-592d-4511-91b3-74bfea7a75ad","Type":"ContainerDied","Data":"bd95fc8aa0f37156a0566cbe796ee8441a3ec741f546deaf6583ec141d00155b"}
Sep 16 14:07:46.570672 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.570628    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"
Sep 16 14:07:46.579468 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.579448    2548 scope.go:117] "RemoveContainer" containerID="e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6"
Sep 16 14:07:46.579776 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:07:46.579751    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6\": container with ID starting with e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6 not found: ID does not exist" containerID="e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6"
Sep 16 14:07:46.579858 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.579789    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6"} err="failed to get container status \"e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6\": rpc error: code = NotFound desc = could not find container \"e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6\": container with ID starting with e5caaa21ef2713921ec6b9ea829e6174d9300e8698de538524d44e2441408cf6 not found: ID does not exist"
Sep 16 14:07:46.579858 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.579813    2548 scope.go:117] "RemoveContainer" containerID="e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab"
Sep 16 14:07:46.589193 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.589178    2548 scope.go:117] "RemoveContainer" containerID="e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab"
Sep 16 14:07:46.589476 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:07:46.589458    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab\": container with ID starting with e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab not found: ID does not exist" containerID="e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab"
Sep 16 14:07:46.589538 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.589481    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab"} err="failed to get container status \"e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab\": rpc error: code = NotFound desc = could not find container \"e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab\": container with ID starting with e13ccaabd21856b749c8205d4b750705915f6f467396b8e975f9a033972233ab not found: ID does not exist"
Sep 16 14:07:46.617370 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.617348    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"]
Sep 16 14:07:46.621823 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.621801    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-19203d-fcbd89854-6ffff"]
Sep 16 14:07:46.645088 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.645068    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"]
Sep 16 14:07:46.649757 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:46.649732    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-19203d-576779b577-4r2qw"]
Sep 16 14:07:47.970403 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:47.970372    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247c2435-613e-41b5-8f6b-6b80f071081e" path="/var/lib/kubelet/pods/247c2435-613e-41b5-8f6b-6b80f071081e/volumes"
Sep 16 14:07:47.970765 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:47.970721    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ba6caa-592d-4511-91b3-74bfea7a75ad" path="/var/lib/kubelet/pods/87ba6caa-592d-4511-91b3-74bfea7a75ad/volumes"
Sep 16 14:07:47.971042 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:47.971029    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5b971f-e015-4bb7-9de6-71ef9381b294" path="/var/lib/kubelet/pods/9d5b971f-e015-4bb7-9de6-71ef9381b294/volumes"
Sep 16 14:07:53.991435 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:53.991407    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"]
Sep 16 14:07:53.991800 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:53.991580    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy" containerID="cri-o://f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e" gracePeriod=30
Sep 16 14:07:55.630811 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:55.630771    2548 patch_prober.go:28] interesting pod/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:07:55.631162 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:55.630849    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:07:57.924607 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:57.924576    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 14:07:57.925897 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:57.925875    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 14:07:59.261212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.261187    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 14:07:59.363921 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.363834    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istiod-ca-cert\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.363921 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.363873    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-certs\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.363921 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.363898    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9mwv\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-kube-api-access-w9mwv\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.364188 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.363924    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-data\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.364188 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.363971    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-envoy\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.364188 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364002    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-podinfo\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.364188 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364049    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-socket\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.364188 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364080    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-token\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.364188 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364116    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-credential-socket\") pod \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\" (UID: \"431d1ed2-916a-40fd-b8ea-a506bdf4a668\") "
Sep 16 14:07:59.364188 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364130    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:07:59.364620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364333    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:07:59.364620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364365    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-data" (OuterVolumeSpecName: "istio-data") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:07:59.364620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364499    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:07:59.364620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364546    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.364620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364566    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.364620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364579    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.364946 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.364644    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:07:59.366141 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.366117    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-kube-api-access-w9mwv" (OuterVolumeSpecName: "kube-api-access-w9mwv") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "kube-api-access-w9mwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:07:59.366217 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.366146    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:07:59.366543 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.366522    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:07:59.366658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.366636    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-token" (OuterVolumeSpecName: "istio-token") pod "431d1ed2-916a-40fd-b8ea-a506bdf4a668" (UID: "431d1ed2-916a-40fd-b8ea-a506bdf4a668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:07:59.465367 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.465331    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.465367 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.465362    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9mwv\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-kube-api-access-w9mwv\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.465558 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.465379    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.465558 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.465391    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.465558 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.465402    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431d1ed2-916a-40fd-b8ea-a506bdf4a668-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.465558 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.465413    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431d1ed2-916a-40fd-b8ea-a506bdf4a668-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:07:59.644337 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.644303    2548 generic.go:356] "Generic (PLEG): container finished" podID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerID="f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e" exitCode=0
Sep 16 14:07:59.644492 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.644354    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"
Sep 16 14:07:59.644492 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.644385    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" event={"ID":"431d1ed2-916a-40fd-b8ea-a506bdf4a668","Type":"ContainerDied","Data":"f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e"}
Sep 16 14:07:59.644492 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.644440    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj" event={"ID":"431d1ed2-916a-40fd-b8ea-a506bdf4a668","Type":"ContainerDied","Data":"6d71dca0b32b7331f9200ab902c72d7a52b2ab89b1bbac78297ac5e97a7cbd5e"}
Sep 16 14:07:59.644492 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.644462    2548 scope.go:117] "RemoveContainer" containerID="f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e"
Sep 16 14:07:59.654420 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.654401    2548 scope.go:117] "RemoveContainer" containerID="5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9"
Sep 16 14:07:59.662189 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.662173    2548 scope.go:117] "RemoveContainer" containerID="f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e"
Sep 16 14:07:59.662503 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:07:59.662484    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e\": container with ID starting with f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e not found: ID does not exist" containerID="f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e"
Sep 16 14:07:59.662575 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.662509    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e"} err="failed to get container status \"f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e\": rpc error: code = NotFound desc = could not find container \"f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e\": container with ID starting with f510a7b92eb4143be8130d7483408d894111b099ab50b1e1405e6401907acc9e not found: ID does not exist"
Sep 16 14:07:59.662575 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.662526    2548 scope.go:117] "RemoveContainer" containerID="5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9"
Sep 16 14:07:59.662778 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:07:59.662763    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9\": container with ID starting with 5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9 not found: ID does not exist" containerID="5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9"
Sep 16 14:07:59.662816 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.662783    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9"} err="failed to get container status \"5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9\": rpc error: code = NotFound desc = could not find container \"5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9\": container with ID starting with 5624a437a24889d1d1961dcc239800d0b1452e51bfe544ef94d1d77f11cabdf9 not found: ID does not exist"
Sep 16 14:07:59.670625 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.670604    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"]
Sep 16 14:07:59.676087 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.676062    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-19203d-openshift-default-7679c567f4-5pchj"]
Sep 16 14:07:59.970081 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:07:59.970002    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" path="/var/lib/kubelet/pods/431d1ed2-916a-40fd-b8ea-a506bdf4a668/volumes"
Sep 16 14:08:02.332385 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332341    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"]
Sep 16 14:08:02.332772 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332749    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy"
Sep 16 14:08:02.332772 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332760    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy"
Sep 16 14:08:02.332859 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332775    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5b971f-e015-4bb7-9de6-71ef9381b294" containerName="payload-pre-processing"
Sep 16 14:08:02.332859 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332781    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5b971f-e015-4bb7-9de6-71ef9381b294" containerName="payload-pre-processing"
Sep 16 14:08:02.332859 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332795    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ba6caa-592d-4511-91b3-74bfea7a75ad" containerName="payload-processing"
Sep 16 14:08:02.332859 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332801    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ba6caa-592d-4511-91b3-74bfea7a75ad" containerName="payload-processing"
Sep 16 14:08:02.332859 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332809    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy"
Sep 16 14:08:02.332859 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332815    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy"
Sep 16 14:08:02.333039 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332871    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ba6caa-592d-4511-91b3-74bfea7a75ad" containerName="payload-processing"
Sep 16 14:08:02.333039 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332880    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy"
Sep 16 14:08:02.333039 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.332887    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5b971f-e015-4bb7-9de6-71ef9381b294" containerName="payload-pre-processing"
Sep 16 14:08:02.337308 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.337287    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"
Sep 16 14:08:02.339835 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.339803    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:08:02.339835 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.339816    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:08:02.347037 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.347010    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"]
Sep 16 14:08:02.498007 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.497967    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d1ea16-c1c9-4de2-922f-be7f28cf73ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6\" (UID: \"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"
Sep 16 14:08:02.598991 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.598893    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d1ea16-c1c9-4de2-922f-be7f28cf73ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6\" (UID: \"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"
Sep 16 14:08:02.599554 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.599531    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d1ea16-c1c9-4de2-922f-be7f28cf73ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6\" (UID: \"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"
Sep 16 14:08:02.650949 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.650914    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"
Sep 16 14:08:02.937882 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:02.937853    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"]
Sep 16 14:08:02.939409 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:08:02.939377    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d1ea16_c1c9_4de2_922f_be7f28cf73ad.slice/crio-94fa33cea62bd243d96aebace9f5dc82efbf0c954a9a0a1bb7f0620c0f1f4c43 WatchSource:0}: Error finding container 94fa33cea62bd243d96aebace9f5dc82efbf0c954a9a0a1bb7f0620c0f1f4c43: Status 404 returned error can't find the container with id 94fa33cea62bd243d96aebace9f5dc82efbf0c954a9a0a1bb7f0620c0f1f4c43
Sep 16 14:08:03.670955 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:03.670916    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6" event={"ID":"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad","Type":"ContainerStarted","Data":"381c1d3aa139b957ba0a887a2823343fb07a2cbbe2d317480e3cd11bddf8cad6"}
Sep 16 14:08:03.671355 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:03.670982    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6" event={"ID":"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad","Type":"ContainerStarted","Data":"94fa33cea62bd243d96aebace9f5dc82efbf0c954a9a0a1bb7f0620c0f1f4c43"}
Sep 16 14:08:03.691949 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:03.691901    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6" podStartSLOduration=1.6918820270000001 podStartE2EDuration="1.691882027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:08:03.690181013 +0000 UTC m=+1506.317662386" watchObservedRunningTime="2026-09-16 14:08:03.691882027 +0000 UTC m=+1506.319363403"
Sep 16 14:08:04.484708 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.484674    2548 patch_prober.go:28] interesting pod/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 14:08:04.485676 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.484750    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 14:08:04.677825 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.677793    2548 generic.go:356] "Generic (PLEG): container finished" podID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerID="271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf" exitCode=0
Sep 16 14:08:04.678207 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.677866    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" event={"ID":"83cf247a-4735-4479-8cd6-4f6a6938e306","Type":"ContainerDied","Data":"271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf"}
Sep 16 14:08:04.678352 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.678328    2548 scope.go:117] "RemoveContainer" containerID="271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf"
Sep 16 14:08:04.679685 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.679665    2548 generic.go:356] "Generic (PLEG): container finished" podID="f8d1ea16-c1c9-4de2-922f-be7f28cf73ad" containerID="381c1d3aa139b957ba0a887a2823343fb07a2cbbe2d317480e3cd11bddf8cad6" exitCode=0
Sep 16 14:08:04.679772 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.679672    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:08:04.679772 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.679749    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:08:04.679772 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.679749    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6" event={"ID":"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad","Type":"ContainerDied","Data":"381c1d3aa139b957ba0a887a2823343fb07a2cbbe2d317480e3cd11bddf8cad6"}
Sep 16 14:08:04.679893 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:04.679786    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:08:05.691895 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:05.691824    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" event={"ID":"83cf247a-4735-4479-8cd6-4f6a6938e306","Type":"ContainerStarted","Data":"9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199"}
Sep 16 14:08:05.861989 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:05.861963    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"
Sep 16 14:08:06.035308 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.035223    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d1ea16-c1c9-4de2-922f-be7f28cf73ad-maas-api-service-ca\") pod \"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad\" (UID: \"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad\") "
Sep 16 14:08:06.035565 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.035540    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d1ea16-c1c9-4de2-922f-be7f28cf73ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8d1ea16-c1c9-4de2-922f-be7f28cf73ad" (UID: "f8d1ea16-c1c9-4de2-922f-be7f28cf73ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:08:06.136848 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.136817    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d1ea16-c1c9-4de2-922f-be7f28cf73ad-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:06.351308 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.351191    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:08:06.351482 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.351458    2548 patch_prober.go:28] interesting pod/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:08:06.351527 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.351510    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 14:08:06.726875 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.726843    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"
Sep 16 14:08:06.727232 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.726885    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6" event={"ID":"f8d1ea16-c1c9-4de2-922f-be7f28cf73ad","Type":"ContainerDied","Data":"94fa33cea62bd243d96aebace9f5dc82efbf0c954a9a0a1bb7f0620c0f1f4c43"}
Sep 16 14:08:06.727232 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:06.726977    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94fa33cea62bd243d96aebace9f5dc82efbf0c954a9a0a1bb7f0620c0f1f4c43"
Sep 16 14:08:07.351335 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:07.351302    2548 patch_prober.go:28] interesting pod/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:08:07.351507 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:07.351353    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 14:08:08.350994 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:08.350960    2548 patch_prober.go:28] interesting pod/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:08:08.351440 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:08.351017    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 14:08:09.351908 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:09.351868    2548 patch_prober.go:28] interesting pod/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:08:09.352322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:09.351929    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 14:08:10.351661 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:10.351626    2548 patch_prober.go:28] interesting pod/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 14:08:10.351839 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:10.351678    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 14:08:11.354345 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:11.354318    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:08:11.354709 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:11.354690    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:08:11.355370 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:11.355350    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:08:18.531929 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:18.531889    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"]
Sep 16 14:08:18.535670 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:18.535643    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7f75a3-8dbr6"]
Sep 16 14:08:19.969828 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:19.969793    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d1ea16-c1c9-4de2-922f-be7f28cf73ad" path="/var/lib/kubelet/pods/f8d1ea16-c1c9-4de2-922f-be7f28cf73ad/volumes"
Sep 16 14:08:25.733974 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:25.733939    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"]
Sep 16 14:08:25.734414 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:25.734149    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy" containerID="cri-o://a7744bff4c1c795fe6c56b7f1d668f63f6327a30b3779a83b12f6c2e64f25278" gracePeriod=30
Sep 16 14:08:30.874191 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:30.874158    2548 generic.go:356] "Generic (PLEG): container finished" podID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerID="a7744bff4c1c795fe6c56b7f1d668f63f6327a30b3779a83b12f6c2e64f25278" exitCode=0
Sep 16 14:08:30.874537 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:30.874224    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" event={"ID":"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c","Type":"ContainerDied","Data":"a7744bff4c1c795fe6c56b7f1d668f63f6327a30b3779a83b12f6c2e64f25278"}
Sep 16 14:08:30.874537 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:30.874332    2548 scope.go:117] "RemoveContainer" containerID="ba3d18c136c1141ad1e31bfd3c0c422527e292f62d7a75c46a7d1e9ffe8545a9"
Sep 16 14:08:30.991777 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:30.991750    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:08:31.081650 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081558    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-token\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.081650 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081622    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-certs\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.081650 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081641    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-credential-socket\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.081926 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081689    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6btv\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-kube-api-access-s6btv\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.081926 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081745    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-data\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.081926 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081826    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-envoy\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.081926 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081859    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-socket\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.081926 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081892    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-podinfo\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.082200 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.081930    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istiod-ca-cert\") pod \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\" (UID: \"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c\") "
Sep 16 14:08:31.082782 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.082411    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:31.082782 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.082483    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:31.082782 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.082524    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:31.082782 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.082628    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:31.082782 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.082568    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:08:31.084225 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.084187    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:08:31.084515 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.084464    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:08:31.084659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.084631    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-kube-api-access-s6btv" (OuterVolumeSpecName: "kube-api-access-s6btv") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "kube-api-access-s6btv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:08:31.085040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.085017    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" (UID: "7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:31.182784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182756    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.182784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182783    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.182961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182794    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.182961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182803    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.182961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182810    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.182961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182817    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.182961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182824    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.182961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182832    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6btv\" (UniqueName: \"kubernetes.io/projected/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-kube-api-access-s6btv\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.182961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.182841    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:31.883486 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.883456    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"
Sep 16 14:08:31.883956 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.883491    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn" event={"ID":"7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c","Type":"ContainerDied","Data":"7bcb253fa633adea35e8af358b10cf43c81a73571a428131fbfb685b036bfdc4"}
Sep 16 14:08:31.883956 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.883567    2548 scope.go:117] "RemoveContainer" containerID="a7744bff4c1c795fe6c56b7f1d668f63f6327a30b3779a83b12f6c2e64f25278"
Sep 16 14:08:31.909301 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.909275    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"]
Sep 16 14:08:31.915988 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.915965    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7f75a3-openshift-default-7bf5ddd746-q94cn"]
Sep 16 14:08:31.970079 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:31.970050    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" path="/var/lib/kubelet/pods/7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c/volumes"
Sep 16 14:08:32.546683 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:32.546650    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"]
Sep 16 14:08:47.565546 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.565484    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7" podUID="8a4d11db-0ff6-43f0-ac17-d65b31976ba0" containerName="main" containerID="cri-o://58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1" gracePeriod=45
Sep 16 14:08:47.817179 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.817122    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:08:47.944240 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944209    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tls-certs\") pod \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") "
Sep 16 14:08:47.944418 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944272    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-model-cache\") pod \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") "
Sep 16 14:08:47.944418 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944299    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-dshm\") pod \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") "
Sep 16 14:08:47.944418 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944319    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tmp-dir\") pod \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") "
Sep 16 14:08:47.944418 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944341    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-home\") pod \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") "
Sep 16 14:08:47.944622 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944417    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db5z7\" (UniqueName: \"kubernetes.io/projected/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-kube-api-access-db5z7\") pod \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\" (UID: \"8a4d11db-0ff6-43f0-ac17-d65b31976ba0\") "
Sep 16 14:08:47.944685 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944618    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a4d11db-0ff6-43f0-ac17-d65b31976ba0" (UID: "8a4d11db-0ff6-43f0-ac17-d65b31976ba0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:47.944685 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944633    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a4d11db-0ff6-43f0-ac17-d65b31976ba0" (UID: "8a4d11db-0ff6-43f0-ac17-d65b31976ba0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:47.944792 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944719    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-home" (OuterVolumeSpecName: "home") pod "8a4d11db-0ff6-43f0-ac17-d65b31976ba0" (UID: "8a4d11db-0ff6-43f0-ac17-d65b31976ba0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:47.944860 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944833    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:47.944915 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944868    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:47.944915 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.944881    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:47.946566 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.946536    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-dshm" (OuterVolumeSpecName: "dshm") pod "8a4d11db-0ff6-43f0-ac17-d65b31976ba0" (UID: "8a4d11db-0ff6-43f0-ac17-d65b31976ba0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:47.946566 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.946558    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a4d11db-0ff6-43f0-ac17-d65b31976ba0" (UID: "8a4d11db-0ff6-43f0-ac17-d65b31976ba0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:08:47.946566 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.946540    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-kube-api-access-db5z7" (OuterVolumeSpecName: "kube-api-access-db5z7") pod "8a4d11db-0ff6-43f0-ac17-d65b31976ba0" (UID: "8a4d11db-0ff6-43f0-ac17-d65b31976ba0"). InnerVolumeSpecName "kube-api-access-db5z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:08:47.972524 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.972501    2548 generic.go:356] "Generic (PLEG): container finished" podID="8a4d11db-0ff6-43f0-ac17-d65b31976ba0" containerID="58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1" exitCode=0
Sep 16 14:08:47.972629 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.972541    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"
Sep 16 14:08:47.972629 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.972571    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7" event={"ID":"8a4d11db-0ff6-43f0-ac17-d65b31976ba0","Type":"ContainerDied","Data":"58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1"}
Sep 16 14:08:47.972629 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.972598    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7" event={"ID":"8a4d11db-0ff6-43f0-ac17-d65b31976ba0","Type":"ContainerDied","Data":"3b64b44e2c0bd5ca50a983a5b625fe5360c2bdc2217119d805653a65d690f653"}
Sep 16 14:08:47.972629 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.972613    2548 scope.go:117] "RemoveContainer" containerID="58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1"
Sep 16 14:08:47.982365 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.982347    2548 scope.go:117] "RemoveContainer" containerID="58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1"
Sep 16 14:08:47.982740 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:08:47.982711    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1\": container with ID starting with 58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1 not found: ID does not exist" containerID="58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1"
Sep 16 14:08:47.982877 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:47.982751    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1"} err="failed to get container status \"58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1\": rpc error: code = NotFound desc = could not find container \"58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1\": container with ID starting with 58c9aff9f14e0e22d2e4ca64f1342be7dc1e435368b1cf309c6c001aeafe78a1 not found: ID does not exist"
Sep 16 14:08:48.001100 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:48.001081    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"]
Sep 16 14:08:48.002936 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:48.002916    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/auth-test-model-46224d-kserve-564fc9ddc-jfwf7"]
Sep 16 14:08:48.045498 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:48.045476    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db5z7\" (UniqueName: \"kubernetes.io/projected/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-kube-api-access-db5z7\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:48.045498 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:48.045497    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:48.045611 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:48.045506    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a4d11db-0ff6-43f0-ac17-d65b31976ba0-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:49.060199 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:49.060164    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"]
Sep 16 14:08:49.060673 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:49.060448    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy" containerID="cri-o://9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199" gracePeriod=30
Sep 16 14:08:49.969927 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:49.969894    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a4d11db-0ff6-43f0-ac17-d65b31976ba0" path="/var/lib/kubelet/pods/8a4d11db-0ff6-43f0-ac17-d65b31976ba0/volumes"
Sep 16 14:08:54.322927 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.322867    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:08:54.400374 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400343    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vttw2\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-kube-api-access-vttw2\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400528 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400454    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-data\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400528 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400483    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-certs\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400528 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400514    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-podinfo\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400700 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400540    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-token\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400700 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400565    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83cf247a-4735-4479-8cd6-4f6a6938e306-istiod-ca-cert\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400700 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400590    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-credential-socket\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400700 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400657    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-socket\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400904 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400701    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-envoy\") pod \"83cf247a-4735-4479-8cd6-4f6a6938e306\" (UID: \"83cf247a-4735-4479-8cd6-4f6a6938e306\") "
Sep 16 14:08:54.400904 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400767    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:54.401006 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400951    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:54.401006 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.400987    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:54.401115 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.401054    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-data" (OuterVolumeSpecName: "istio-data") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:54.401115 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.401052    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cf247a-4735-4479-8cd6-4f6a6938e306-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:08:54.401115 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.401087    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:54.401115 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.401104    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:54.401367 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.401122    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:54.402548 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.402514    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-kube-api-access-vttw2" (OuterVolumeSpecName: "kube-api-access-vttw2") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "kube-api-access-vttw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:08:54.403059 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.403038    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:08:54.403059 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.403062    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:08:54.403230 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.403090    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-token" (OuterVolumeSpecName: "istio-token") pod "83cf247a-4735-4479-8cd6-4f6a6938e306" (UID: "83cf247a-4735-4479-8cd6-4f6a6938e306"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:08:54.501574 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.501547    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:54.501574 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.501568    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:54.501574 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.501577    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:54.501760 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.501587    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83cf247a-4735-4479-8cd6-4f6a6938e306-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:54.501760 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.501594    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83cf247a-4735-4479-8cd6-4f6a6938e306-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:54.501760 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:54.501602    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttw2\" (UniqueName: \"kubernetes.io/projected/83cf247a-4735-4479-8cd6-4f6a6938e306-kube-api-access-vttw2\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:08:55.012628 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.012593    2548 generic.go:356] "Generic (PLEG): container finished" podID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerID="9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199" exitCode=0
Sep 16 14:08:55.012799 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.012640    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"
Sep 16 14:08:55.012799 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.012646    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" event={"ID":"83cf247a-4735-4479-8cd6-4f6a6938e306","Type":"ContainerDied","Data":"9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199"}
Sep 16 14:08:55.012799 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.012688    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66" event={"ID":"83cf247a-4735-4479-8cd6-4f6a6938e306","Type":"ContainerDied","Data":"0c6620f14165dff72c6d6449f6064f74733aef9c1f8b4a1e4441bf4eef0b26d8"}
Sep 16 14:08:55.012799 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.012709    2548 scope.go:117] "RemoveContainer" containerID="9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199"
Sep 16 14:08:55.021945 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.021927    2548 scope.go:117] "RemoveContainer" containerID="271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf"
Sep 16 14:08:55.029792 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.029777    2548 scope.go:117] "RemoveContainer" containerID="9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199"
Sep 16 14:08:55.030043 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:08:55.030023    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199\": container with ID starting with 9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199 not found: ID does not exist" containerID="9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199"
Sep 16 14:08:55.030118 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.030055    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199"} err="failed to get container status \"9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199\": rpc error: code = NotFound desc = could not find container \"9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199\": container with ID starting with 9513884358d6db642224e830094b11d1eb4e661c410e92aa2952e88879366199 not found: ID does not exist"
Sep 16 14:08:55.030118 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.030077    2548 scope.go:117] "RemoveContainer" containerID="271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf"
Sep 16 14:08:55.030377 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:08:55.030352    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf\": container with ID starting with 271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf not found: ID does not exist" containerID="271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf"
Sep 16 14:08:55.030429 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.030377    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf"} err="failed to get container status \"271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf\": rpc error: code = NotFound desc = could not find container \"271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf\": container with ID starting with 271c04f3966d6d2f72e1d7e3340bba7edaea52bc6bc9310ff9e4920425499ddf not found: ID does not exist"
Sep 16 14:08:55.039738 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.039718    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"]
Sep 16 14:08:55.045961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.045940    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8a4447-openshift-default-648cdb974-nzg66"]
Sep 16 14:08:55.969626 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:55.969594    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" path="/var/lib/kubelet/pods/83cf247a-4735-4479-8cd6-4f6a6938e306/volumes"
Sep 16 14:08:58.086289 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:08:58.084012    2548 scope.go:117] "RemoveContainer" containerID="adcf54b8dc521efdd5d3b782791bde69a67c82f97de1b5c021c154a0ad45efe1"
Sep 16 14:09:00.138543 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.138509    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"]
Sep 16 14:09:00.139145 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.139112    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2" podUID="098a8c2e-e69c-4432-b1be-f56a3529e5c9" containerName="payload-pre-processing" containerID="cri-o://35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8" gracePeriod=30
Sep 16 14:09:00.406462 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.406436    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:09:00.453053 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.453014    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098a8c2e-e69c-4432-b1be-f56a3529e5c9-plugins-config-volume\") pod \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\" (UID: \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\") "
Sep 16 14:09:00.453299 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.453163    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djxth\" (UniqueName: \"kubernetes.io/projected/098a8c2e-e69c-4432-b1be-f56a3529e5c9-kube-api-access-djxth\") pod \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\" (UID: \"098a8c2e-e69c-4432-b1be-f56a3529e5c9\") "
Sep 16 14:09:00.453798 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.453582    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098a8c2e-e69c-4432-b1be-f56a3529e5c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "098a8c2e-e69c-4432-b1be-f56a3529e5c9" (UID: "098a8c2e-e69c-4432-b1be-f56a3529e5c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:09:00.456370 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.456333    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098a8c2e-e69c-4432-b1be-f56a3529e5c9-kube-api-access-djxth" (OuterVolumeSpecName: "kube-api-access-djxth") pod "098a8c2e-e69c-4432-b1be-f56a3529e5c9" (UID: "098a8c2e-e69c-4432-b1be-f56a3529e5c9"). InnerVolumeSpecName "kube-api-access-djxth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:09:00.554676 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.554641    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/098a8c2e-e69c-4432-b1be-f56a3529e5c9-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:00.554676 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:00.554672    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djxth\" (UniqueName: \"kubernetes.io/projected/098a8c2e-e69c-4432-b1be-f56a3529e5c9-kube-api-access-djxth\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:01.048484 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.048449    2548 generic.go:356] "Generic (PLEG): container finished" podID="098a8c2e-e69c-4432-b1be-f56a3529e5c9" containerID="35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8" exitCode=0
Sep 16 14:09:01.048658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.048491    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"
Sep 16 14:09:01.048658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.048505    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2" event={"ID":"098a8c2e-e69c-4432-b1be-f56a3529e5c9","Type":"ContainerDied","Data":"35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8"}
Sep 16 14:09:01.048658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.048548    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2" event={"ID":"098a8c2e-e69c-4432-b1be-f56a3529e5c9","Type":"ContainerDied","Data":"be626c1dde57473a8652ad2c0c5efaeca065126cf537fbda0f635a7e943c5bde"}
Sep 16 14:09:01.048658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.048570    2548 scope.go:117] "RemoveContainer" containerID="35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8"
Sep 16 14:09:01.057544 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.057519    2548 scope.go:117] "RemoveContainer" containerID="35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8"
Sep 16 14:09:01.057799 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:09:01.057780    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8\": container with ID starting with 35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8 not found: ID does not exist" containerID="35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8"
Sep 16 14:09:01.057879 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.057804    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8"} err="failed to get container status \"35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8\": rpc error: code = NotFound desc = could not find container \"35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8\": container with ID starting with 35803c774c230360d61d7f3cb6bbda96f5b9fb2cac13addd25ba30db66bcb0b8 not found: ID does not exist"
Sep 16 14:09:01.071522 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.071497    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"]
Sep 16 14:09:01.079349 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.079327    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-740a25-55876c8d98-hcdf2"]
Sep 16 14:09:01.970184 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:01.970150    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098a8c2e-e69c-4432-b1be-f56a3529e5c9" path="/var/lib/kubelet/pods/098a8c2e-e69c-4432-b1be-f56a3529e5c9/volumes"
Sep 16 14:09:07.316747 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:07.316716    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"]
Sep 16 14:09:07.317241 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:07.316944    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy" containerID="cri-o://b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3" gracePeriod=30
Sep 16 14:09:12.558399 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.558371    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:09:12.672543 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672501    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-certs\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.672719 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672598    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-envoy\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.672719 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672615    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-data\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.672719 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672640    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96eb6564-246d-46fb-b204-8fdff138a99b-istiod-ca-cert\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.672719 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672658    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-socket\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.672719 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672680    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-istio-token\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.672719 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672712    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-credential-socket\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.673053 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672746    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96eb6564-246d-46fb-b204-8fdff138a99b-istio-podinfo\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.673053 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672776    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5f4n\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-kube-api-access-d5f4n\") pod \"96eb6564-246d-46fb-b204-8fdff138a99b\" (UID: \"96eb6564-246d-46fb-b204-8fdff138a99b\") "
Sep 16 14:09:12.673053 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672793    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:12.673053 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.672905    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-data" (OuterVolumeSpecName: "istio-data") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:12.673287 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.673086    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:12.673287 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.673136    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:12.673287 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.673096    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:12.673287 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.673210    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:12.673474 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.673339    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96eb6564-246d-46fb-b204-8fdff138a99b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:09:12.674978 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.674949    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-istio-token" (OuterVolumeSpecName: "istio-token") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:09:12.675080 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.675057    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:12.675322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.675302    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96eb6564-246d-46fb-b204-8fdff138a99b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:09:12.675423 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.675406    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-kube-api-access-d5f4n" (OuterVolumeSpecName: "kube-api-access-d5f4n") pod "96eb6564-246d-46fb-b204-8fdff138a99b" (UID: "96eb6564-246d-46fb-b204-8fdff138a99b"). InnerVolumeSpecName "kube-api-access-d5f4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:09:12.773637 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.773612    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96eb6564-246d-46fb-b204-8fdff138a99b-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:12.773637 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.773636    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:12.773637 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.773646    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:12.773847 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.773654    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:12.773847 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.773663    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96eb6564-246d-46fb-b204-8fdff138a99b-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:12.773847 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.773673    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5f4n\" (UniqueName: \"kubernetes.io/projected/96eb6564-246d-46fb-b204-8fdff138a99b-kube-api-access-d5f4n\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:12.773847 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:12.773681    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96eb6564-246d-46fb-b204-8fdff138a99b-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:13.113746 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.113659    2548 generic.go:356] "Generic (PLEG): container finished" podID="96eb6564-246d-46fb-b204-8fdff138a99b" containerID="b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3" exitCode=0
Sep 16 14:09:13.113746 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.113700    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"
Sep 16 14:09:13.113923 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.113746    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" event={"ID":"96eb6564-246d-46fb-b204-8fdff138a99b","Type":"ContainerDied","Data":"b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3"}
Sep 16 14:09:13.113923 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.113801    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6" event={"ID":"96eb6564-246d-46fb-b204-8fdff138a99b","Type":"ContainerDied","Data":"cf9a1e17cc3f71e9d430e07db0a785fb13d62a5c746d17cd1197089030f3f31c"}
Sep 16 14:09:13.113923 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.113826    2548 scope.go:117] "RemoveContainer" containerID="b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3"
Sep 16 14:09:13.123013 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.122996    2548 scope.go:117] "RemoveContainer" containerID="33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3"
Sep 16 14:09:13.130480 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.130463    2548 scope.go:117] "RemoveContainer" containerID="b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3"
Sep 16 14:09:13.130719 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:09:13.130701    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3\": container with ID starting with b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3 not found: ID does not exist" containerID="b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3"
Sep 16 14:09:13.130762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.130728    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3"} err="failed to get container status \"b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3\": rpc error: code = NotFound desc = could not find container \"b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3\": container with ID starting with b3fc4112d8c2f5d5125e55d35e2e43393df07cb54318ebe6b801c021e66ac6e3 not found: ID does not exist"
Sep 16 14:09:13.130762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.130746    2548 scope.go:117] "RemoveContainer" containerID="33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3"
Sep 16 14:09:13.130961 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:09:13.130945    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3\": container with ID starting with 33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3 not found: ID does not exist" containerID="33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3"
Sep 16 14:09:13.131003 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.130967    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3"} err="failed to get container status \"33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3\": rpc error: code = NotFound desc = could not find container \"33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3\": container with ID starting with 33b3646937ad94d88cb91374e57c94e09b4c8bd748a42dcbf731c9038b594ca3 not found: ID does not exist"
Sep 16 14:09:13.137144 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.137122    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"]
Sep 16 14:09:13.141505 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.141485    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-740a25-openshift-default-65d85f5496-287p6"]
Sep 16 14:09:13.969607 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:13.969574    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" path="/var/lib/kubelet/pods/96eb6564-246d-46fb-b204-8fdff138a99b/volumes"
Sep 16 14:09:25.385795 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:25.385759    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"]
Sep 16 14:09:25.386421 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:25.386047    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy" containerID="cri-o://3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12" gracePeriod=30
Sep 16 14:09:30.639739 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.639710    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:09:30.741155 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741071    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-credential-socket\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741155 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741111    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0117d88d-499d-4196-badf-6e77bd473b17-istio-podinfo\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741155 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741141    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-data\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741167    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0117d88d-499d-4196-badf-6e77bd473b17-istiod-ca-cert\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741194    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-istio-token\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741224    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-certs\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741277    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-socket\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741307    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-envoy\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741330    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcr2k\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-kube-api-access-fcr2k\") pod \"0117d88d-499d-4196-badf-6e77bd473b17\" (UID: \"0117d88d-499d-4196-badf-6e77bd473b17\") "
Sep 16 14:09:30.741559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741373    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:30.741559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741452    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-data" (OuterVolumeSpecName: "istio-data") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:30.741953 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741611    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:30.741953 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741625    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0117d88d-499d-4196-badf-6e77bd473b17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:09:30.741953 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741778    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:30.741953 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741797    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:30.741953 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741790    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:30.741953 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741812    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:30.741953 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.741824    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0117d88d-499d-4196-badf-6e77bd473b17-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:30.743363 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.743331    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0117d88d-499d-4196-badf-6e77bd473b17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:09:30.743492 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.743449    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-istio-token" (OuterVolumeSpecName: "istio-token") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:09:30.743492 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.743470    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-kube-api-access-fcr2k" (OuterVolumeSpecName: "kube-api-access-fcr2k") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "kube-api-access-fcr2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:09:30.743658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.743638    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0117d88d-499d-4196-badf-6e77bd473b17" (UID: "0117d88d-499d-4196-badf-6e77bd473b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:09:30.842493 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.842469    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:30.842493 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.842491    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcr2k\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-kube-api-access-fcr2k\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:30.842643 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.842501    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0117d88d-499d-4196-badf-6e77bd473b17-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:30.842643 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.842510    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0117d88d-499d-4196-badf-6e77bd473b17-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:30.842643 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:30.842519    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0117d88d-499d-4196-badf-6e77bd473b17-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:09:31.208018 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.207985    2548 generic.go:356] "Generic (PLEG): container finished" podID="0117d88d-499d-4196-badf-6e77bd473b17" containerID="3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12" exitCode=0
Sep 16 14:09:31.208186 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.208031    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" event={"ID":"0117d88d-499d-4196-badf-6e77bd473b17","Type":"ContainerDied","Data":"3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12"}
Sep 16 14:09:31.208186 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.208067    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"
Sep 16 14:09:31.208186 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.208092    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq" event={"ID":"0117d88d-499d-4196-badf-6e77bd473b17","Type":"ContainerDied","Data":"59cbf577def504a523a6c588b1d087785e923d3fee7b81490a5493f1d5940804"}
Sep 16 14:09:31.208186 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.208117    2548 scope.go:117] "RemoveContainer" containerID="3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12"
Sep 16 14:09:31.219097 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.219081    2548 scope.go:117] "RemoveContainer" containerID="1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0"
Sep 16 14:09:31.227884 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.227866    2548 scope.go:117] "RemoveContainer" containerID="3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12"
Sep 16 14:09:31.228154 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:09:31.228135    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12\": container with ID starting with 3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12 not found: ID does not exist" containerID="3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12"
Sep 16 14:09:31.228198 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.228162    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12"} err="failed to get container status \"3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12\": rpc error: code = NotFound desc = could not find container \"3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12\": container with ID starting with 3a3b9d3f072002d834a42dcdfba7f912f21b1a879345a427083900a6ff748a12 not found: ID does not exist"
Sep 16 14:09:31.228198 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.228179    2548 scope.go:117] "RemoveContainer" containerID="1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0"
Sep 16 14:09:31.228465 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:09:31.228448    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0\": container with ID starting with 1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0 not found: ID does not exist" containerID="1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0"
Sep 16 14:09:31.228517 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.228469    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0"} err="failed to get container status \"1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0\": rpc error: code = NotFound desc = could not find container \"1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0\": container with ID starting with 1c247682403f9bb8b2ac2356d868894b8308f3bcc0a92ace20250b0a18e1bbb0 not found: ID does not exist"
Sep 16 14:09:31.241504 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.241481    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"]
Sep 16 14:09:31.257111 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.257091    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9187f5-openshift-default-6d95985c6-qrqfq"]
Sep 16 14:09:31.970052 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:31.970023    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0117d88d-499d-4196-badf-6e77bd473b17" path="/var/lib/kubelet/pods/0117d88d-499d-4196-badf-6e77bd473b17/volumes"
Sep 16 14:09:54.552958 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.552878    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"]
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553279    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553293    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553304    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553309    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553316    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553322    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553328    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553334    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553343    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553350    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553358    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553363    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553374    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098a8c2e-e69c-4432-b1be-f56a3529e5c9" containerName="payload-pre-processing"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553382    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="098a8c2e-e69c-4432-b1be-f56a3529e5c9" containerName="payload-pre-processing"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553387    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553392    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553398    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a4d11db-0ff6-43f0-ac17-d65b31976ba0" containerName="main"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553402    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a4d11db-0ff6-43f0-ac17-d65b31976ba0" containerName="main"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553415    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d1ea16-c1c9-4de2-922f-be7f28cf73ad" containerName="revoke-keys"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553419    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d1ea16-c1c9-4de2-922f-be7f28cf73ad" containerName="revoke-keys"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553430    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy"
Sep 16 14:09:54.553457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553435    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553488    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a4d11db-0ff6-43f0-ac17-d65b31976ba0" containerName="main"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553494    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553500    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553505    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553511    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d1ea16-c1c9-4de2-922f-be7f28cf73ad" containerName="revoke-keys"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553516    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="83cf247a-4735-4479-8cd6-4f6a6938e306" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553524    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="431d1ed2-916a-40fd-b8ea-a506bdf4a668" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553532    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553537    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="0117d88d-499d-4196-badf-6e77bd473b17" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553543    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1a1cda-8d33-40ac-b1ed-7a29f01aec7c" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553547    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="96eb6564-246d-46fb-b204-8fdff138a99b" containerName="istio-proxy"
Sep 16 14:09:54.554635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.553554    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="098a8c2e-e69c-4432-b1be-f56a3529e5c9" containerName="payload-pre-processing"
Sep 16 14:09:54.558210 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.558186    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.563173 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.563143    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"kube-root-ca.crt\""
Sep 16 14:09:54.563315 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.563172    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"openshift-service-ca.crt\""
Sep 16 14:09:54.563315 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.563172    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"rate-test-model-46224d-kserve-self-signed-certs\""
Sep 16 14:09:54.563315 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.563272    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-46224d\"/\"default-dockercfg-7g6qm\""
Sep 16 14:09:54.573077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.573054    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"]
Sep 16 14:09:54.658306 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.658279    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84948f99-6565-47cb-bd86-e0f68999da01-tls-certs\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.658306 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.658307    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-dshm\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.658490 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.658335    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8nrp\" (UniqueName: \"kubernetes.io/projected/84948f99-6565-47cb-bd86-e0f68999da01-kube-api-access-q8nrp\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.658490 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.658413    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-tmp-dir\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.658490 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.658468    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-model-cache\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.658645 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.658573    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-home\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759158 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759123    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-home\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759349 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759175    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84948f99-6565-47cb-bd86-e0f68999da01-tls-certs\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759349 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759202    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-dshm\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759349 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759238    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8nrp\" (UniqueName: \"kubernetes.io/projected/84948f99-6565-47cb-bd86-e0f68999da01-kube-api-access-q8nrp\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759349 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759306    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-tmp-dir\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759548 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759359    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-model-cache\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759652 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759627    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-home\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759728 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759653    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-model-cache\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.759784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.759734    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-tmp-dir\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.761390 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.761370    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-dshm\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.761702 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.761684    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84948f99-6565-47cb-bd86-e0f68999da01-tls-certs\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.770211 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.770192    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8nrp\" (UniqueName: \"kubernetes.io/projected/84948f99-6565-47cb-bd86-e0f68999da01-kube-api-access-q8nrp\") pod \"rate-test-model-46224d-kserve-79c6585859-7qllm\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") " pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:54.867614 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:54.867586    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:09:55.139917 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:55.139882    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"]
Sep 16 14:09:55.142523 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:09:55.142497    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84948f99_6565_47cb_bd86_e0f68999da01.slice/crio-e4cbf2c5487c8b02daeb9c03bd93b963e87e52e2609c1ebcc48bf5108ee42195 WatchSource:0}: Error finding container e4cbf2c5487c8b02daeb9c03bd93b963e87e52e2609c1ebcc48bf5108ee42195: Status 404 returned error can't find the container with id e4cbf2c5487c8b02daeb9c03bd93b963e87e52e2609c1ebcc48bf5108ee42195
Sep 16 14:09:55.328620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:55.328586    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm" event={"ID":"84948f99-6565-47cb-bd86-e0f68999da01","Type":"ContainerStarted","Data":"9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b"}
Sep 16 14:09:55.328784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:55.328628    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm" event={"ID":"84948f99-6565-47cb-bd86-e0f68999da01","Type":"ContainerStarted","Data":"e4cbf2c5487c8b02daeb9c03bd93b963e87e52e2609c1ebcc48bf5108ee42195"}
Sep 16 14:09:55.348944 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:09:55.348905    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm" podStartSLOduration=1.348891302 podStartE2EDuration="1.348891302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:09:55.34728206 +0000 UTC m=+1617.974763435" watchObservedRunningTime="2026-09-16 14:09:55.348891302 +0000 UTC m=+1617.976372675"
Sep 16 14:10:04.868417 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:10:04.868370    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:10:04.868417 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:10:04.868434    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:10:04.881512 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:10:04.881483    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:10:05.391055 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:10:05.391027    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:10:58.143540 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:10:58.143514    2548 scope.go:117] "RemoveContainer" containerID="277bc0cda300e0ced46d37a16203331b1d92a2c446c6bd7cc79d8ff3a793dc96"
Sep 16 14:11:03.389598 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:03.389566    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"]
Sep 16 14:11:11.151563 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.151525    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"]
Sep 16 14:11:11.155081 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.155063    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:11.157900 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.157881    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cabdc2-dockercfg-9bb7v\""
Sep 16 14:11:11.157984 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.157882    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cabdc2\""
Sep 16 14:11:11.165626 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.165603    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"]
Sep 16 14:11:11.253362 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.253295    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87bc362-49d2-4131-ae29-07abab55883d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd\" (UID: \"d87bc362-49d2-4131-ae29-07abab55883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:11.253572 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.253472    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bvcn\" (UniqueName: \"kubernetes.io/projected/d87bc362-49d2-4131-ae29-07abab55883d-kube-api-access-4bvcn\") pod \"payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd\" (UID: \"d87bc362-49d2-4131-ae29-07abab55883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:11.354132 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.354098    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87bc362-49d2-4131-ae29-07abab55883d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd\" (UID: \"d87bc362-49d2-4131-ae29-07abab55883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:11.354322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.354161    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bvcn\" (UniqueName: \"kubernetes.io/projected/d87bc362-49d2-4131-ae29-07abab55883d-kube-api-access-4bvcn\") pod \"payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd\" (UID: \"d87bc362-49d2-4131-ae29-07abab55883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:11.354798 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.354779    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87bc362-49d2-4131-ae29-07abab55883d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd\" (UID: \"d87bc362-49d2-4131-ae29-07abab55883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:11.363876 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.363845    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bvcn\" (UniqueName: \"kubernetes.io/projected/d87bc362-49d2-4131-ae29-07abab55883d-kube-api-access-4bvcn\") pod \"payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd\" (UID: \"d87bc362-49d2-4131-ae29-07abab55883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:11.466288 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.466217    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:11.741602 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:11.737549    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"]
Sep 16 14:11:11.743209 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:11:11.743180    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87bc362_49d2_4131_ae29_07abab55883d.slice/crio-35428dc3cc84252f2ae4516a77fcce31226c3b4f8e3ade49ac2e77fc6fdaf286 WatchSource:0}: Error finding container 35428dc3cc84252f2ae4516a77fcce31226c3b4f8e3ade49ac2e77fc6fdaf286: Status 404 returned error can't find the container with id 35428dc3cc84252f2ae4516a77fcce31226c3b4f8e3ade49ac2e77fc6fdaf286
Sep 16 14:11:12.718079 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:12.718036    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd" event={"ID":"d87bc362-49d2-4131-ae29-07abab55883d","Type":"ContainerStarted","Data":"6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86"}
Sep 16 14:11:12.718473 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:12.718091    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd" event={"ID":"d87bc362-49d2-4131-ae29-07abab55883d","Type":"ContainerStarted","Data":"35428dc3cc84252f2ae4516a77fcce31226c3b4f8e3ade49ac2e77fc6fdaf286"}
Sep 16 14:11:12.718473 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:12.718126    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:12.735309 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:12.735262    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd" podStartSLOduration=1.735229409 podStartE2EDuration="1.735229409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:12.7347854 +0000 UTC m=+1695.362266774" watchObservedRunningTime="2026-09-16 14:11:12.735229409 +0000 UTC m=+1695.362710785"
Sep 16 14:11:18.408264 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.408175    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm" podUID="84948f99-6565-47cb-bd86-e0f68999da01" containerName="main" containerID="cri-o://9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b" gracePeriod=45
Sep 16 14:11:18.652178 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.652154    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:11:18.721061 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.720992    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-tmp-dir\") pod \"84948f99-6565-47cb-bd86-e0f68999da01\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") "
Sep 16 14:11:18.721061 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721024    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84948f99-6565-47cb-bd86-e0f68999da01-tls-certs\") pod \"84948f99-6565-47cb-bd86-e0f68999da01\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") "
Sep 16 14:11:18.721275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721077    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-dshm\") pod \"84948f99-6565-47cb-bd86-e0f68999da01\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") "
Sep 16 14:11:18.721275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721101    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-home\") pod \"84948f99-6565-47cb-bd86-e0f68999da01\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") "
Sep 16 14:11:18.721275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721124    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-model-cache\") pod \"84948f99-6565-47cb-bd86-e0f68999da01\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") "
Sep 16 14:11:18.721275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721143    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8nrp\" (UniqueName: \"kubernetes.io/projected/84948f99-6565-47cb-bd86-e0f68999da01-kube-api-access-q8nrp\") pod \"84948f99-6565-47cb-bd86-e0f68999da01\" (UID: \"84948f99-6565-47cb-bd86-e0f68999da01\") "
Sep 16 14:11:18.721494 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721241    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84948f99-6565-47cb-bd86-e0f68999da01" (UID: "84948f99-6565-47cb-bd86-e0f68999da01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:11:18.721494 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721401    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-model-cache" (OuterVolumeSpecName: "model-cache") pod "84948f99-6565-47cb-bd86-e0f68999da01" (UID: "84948f99-6565-47cb-bd86-e0f68999da01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:11:18.721494 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721419    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-home" (OuterVolumeSpecName: "home") pod "84948f99-6565-47cb-bd86-e0f68999da01" (UID: "84948f99-6565-47cb-bd86-e0f68999da01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:11:18.721613 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721572    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:11:18.721613 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721588    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:11:18.721613 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.721601    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:11:18.723191 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.723168    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84948f99-6565-47cb-bd86-e0f68999da01-kube-api-access-q8nrp" (OuterVolumeSpecName: "kube-api-access-q8nrp") pod "84948f99-6565-47cb-bd86-e0f68999da01" (UID: "84948f99-6565-47cb-bd86-e0f68999da01"). InnerVolumeSpecName "kube-api-access-q8nrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:11:18.723610 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.723592    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84948f99-6565-47cb-bd86-e0f68999da01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84948f99-6565-47cb-bd86-e0f68999da01" (UID: "84948f99-6565-47cb-bd86-e0f68999da01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:11:18.723662 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.723620    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-dshm" (OuterVolumeSpecName: "dshm") pod "84948f99-6565-47cb-bd86-e0f68999da01" (UID: "84948f99-6565-47cb-bd86-e0f68999da01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:11:18.750018 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.749993    2548 generic.go:356] "Generic (PLEG): container finished" podID="84948f99-6565-47cb-bd86-e0f68999da01" containerID="9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b" exitCode=0
Sep 16 14:11:18.750134 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.750041    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"
Sep 16 14:11:18.750134 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.750074    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm" event={"ID":"84948f99-6565-47cb-bd86-e0f68999da01","Type":"ContainerDied","Data":"9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b"}
Sep 16 14:11:18.750134 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.750111    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm" event={"ID":"84948f99-6565-47cb-bd86-e0f68999da01","Type":"ContainerDied","Data":"e4cbf2c5487c8b02daeb9c03bd93b963e87e52e2609c1ebcc48bf5108ee42195"}
Sep 16 14:11:18.750134 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.750127    2548 scope.go:117] "RemoveContainer" containerID="9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b"
Sep 16 14:11:18.758661 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.758643    2548 scope.go:117] "RemoveContainer" containerID="9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b"
Sep 16 14:11:18.758933 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:11:18.758915    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b\": container with ID starting with 9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b not found: ID does not exist" containerID="9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b"
Sep 16 14:11:18.758987 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.758940    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b"} err="failed to get container status \"9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b\": rpc error: code = NotFound desc = could not find container \"9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b\": container with ID starting with 9ce0808a053fd8a7361784622a325e4d957dfe0ed1bdecf170a6d27bdf5d879b not found: ID does not exist"
Sep 16 14:11:18.775451 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.775430    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"]
Sep 16 14:11:18.780639 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.780619    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-46224d/rate-test-model-46224d-kserve-79c6585859-7qllm"]
Sep 16 14:11:18.822195 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.822175    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84948f99-6565-47cb-bd86-e0f68999da01-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:11:18.822195 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.822193    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84948f99-6565-47cb-bd86-e0f68999da01-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:11:18.822336 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:18.822202    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8nrp\" (UniqueName: \"kubernetes.io/projected/84948f99-6565-47cb-bd86-e0f68999da01-kube-api-access-q8nrp\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:11:19.972017 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:19.971985    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84948f99-6565-47cb-bd86-e0f68999da01" path="/var/lib/kubelet/pods/84948f99-6565-47cb-bd86-e0f68999da01/volumes"
Sep 16 14:11:23.725568 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:23.725532    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:11:40.935778 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:40.935748    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"]
Sep 16 14:11:40.936350 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:40.936328    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84948f99-6565-47cb-bd86-e0f68999da01" containerName="main"
Sep 16 14:11:40.936350 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:40.936347    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="84948f99-6565-47cb-bd86-e0f68999da01" containerName="main"
Sep 16 14:11:40.936513 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:40.936418    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="84948f99-6565-47cb-bd86-e0f68999da01" containerName="main"
Sep 16 14:11:40.939434 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:40.939416    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:40.942387 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:40.942364    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9c8daa-dockercfg-9qwfw\""
Sep 16 14:11:40.942487 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:40.942386    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9c8daa\""
Sep 16 14:11:40.952977 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:40.952955    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"]
Sep 16 14:11:41.023021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.022989    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s447j\" (UniqueName: \"kubernetes.io/projected/04fbd135-6e63-43d1-96c0-686bded782cd-kube-api-access-s447j\") pod \"payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff\" (UID: \"04fbd135-6e63-43d1-96c0-686bded782cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:41.023240 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.023213    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fbd135-6e63-43d1-96c0-686bded782cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff\" (UID: \"04fbd135-6e63-43d1-96c0-686bded782cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:41.123993 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.123960    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s447j\" (UniqueName: \"kubernetes.io/projected/04fbd135-6e63-43d1-96c0-686bded782cd-kube-api-access-s447j\") pod \"payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff\" (UID: \"04fbd135-6e63-43d1-96c0-686bded782cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:41.124178 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.124004    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fbd135-6e63-43d1-96c0-686bded782cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff\" (UID: \"04fbd135-6e63-43d1-96c0-686bded782cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:41.124649 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.124629    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fbd135-6e63-43d1-96c0-686bded782cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff\" (UID: \"04fbd135-6e63-43d1-96c0-686bded782cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:41.133168 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.133146    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s447j\" (UniqueName: \"kubernetes.io/projected/04fbd135-6e63-43d1-96c0-686bded782cd-kube-api-access-s447j\") pod \"payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff\" (UID: \"04fbd135-6e63-43d1-96c0-686bded782cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:41.252167 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.252088    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:41.524731 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.524704    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"]
Sep 16 14:11:41.526591 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:11:41.526562    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fbd135_6e63_43d1_96c0_686bded782cd.slice/crio-2b54e007a78ae786d7ebcabc14d34d161220d3a121808522e7dfc51ad740edb1 WatchSource:0}: Error finding container 2b54e007a78ae786d7ebcabc14d34d161220d3a121808522e7dfc51ad740edb1: Status 404 returned error can't find the container with id 2b54e007a78ae786d7ebcabc14d34d161220d3a121808522e7dfc51ad740edb1
Sep 16 14:11:41.869689 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.869652    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff" event={"ID":"04fbd135-6e63-43d1-96c0-686bded782cd","Type":"ContainerStarted","Data":"2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2"}
Sep 16 14:11:41.869845 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.869717    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff" event={"ID":"04fbd135-6e63-43d1-96c0-686bded782cd","Type":"ContainerStarted","Data":"2b54e007a78ae786d7ebcabc14d34d161220d3a121808522e7dfc51ad740edb1"}
Sep 16 14:11:41.869845 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.869779    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:11:41.890731 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:41.890683    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff" podStartSLOduration=1.890665901 podStartE2EDuration="1.890665901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:41.889224873 +0000 UTC m=+1724.516706260" watchObservedRunningTime="2026-09-16 14:11:41.890665901 +0000 UTC m=+1724.518147277"
Sep 16 14:11:52.876849 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:11:52.876819    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:12:07.538958 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:07.538926    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff_04fbd135-6e63-43d1-96c0-686bded782cd/payload-processing/0.log"
Sep 16 14:12:38.384609 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:38.384578    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd_d87bc362-49d2-4131-ae29-07abab55883d/payload-processing/0.log"
Sep 16 14:12:38.624428 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:38.624393    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff_04fbd135-6e63-43d1-96c0-686bded782cd/payload-processing/0.log"
Sep 16 14:12:39.218951 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.218918    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"]
Sep 16 14:12:39.222400 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.222379    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.225193 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.225162    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-29812a-openshift-default-dockercfg-f6d2j\""
Sep 16 14:12:39.235032 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.235007    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"]
Sep 16 14:12:39.347320 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347281    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-podinfo\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.347490 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347334    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-socket\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.347490 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347356    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.347490 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347373    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-token\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.347620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347528    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-envoy\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.347620 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347575    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-certs\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.347696 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347625    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-data\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.347696 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347662    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-credential-socket\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.347769 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.347699    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-kube-api-access-zswm2\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448423 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448389    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-podinfo\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448873 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448441    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-socket\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448873 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448469    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448873 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448495    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-token\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448873 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448573    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-envoy\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448873 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448608    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-certs\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448873 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448641    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-data\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448873 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448673    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-credential-socket\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.448873 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448710    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-kube-api-access-zswm2\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.449236 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.448933    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-socket\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.449236 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.449011    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-certs\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.449236 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.449086    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-credential-socket\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.449430 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.449238    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-data\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.449734 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.449711    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.451198 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.451177    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-podinfo\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.451307 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.451224    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-envoy\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.457092 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.457071    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-token\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.457300 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.457279    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-kube-api-access-zswm2\") pod \"e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") " pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.533659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.533596    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:39.809104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.809082    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"]
Sep 16 14:12:39.810660 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:12:39.810624    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1b7584_889d_432f_bb61_b1d3b9ab560b.slice/crio-1bb87950a9dd04d85001a6c62addc150c0355ce95ccb69f93dd6180df4c11a2e WatchSource:0}: Error finding container 1bb87950a9dd04d85001a6c62addc150c0355ce95ccb69f93dd6180df4c11a2e: Status 404 returned error can't find the container with id 1bb87950a9dd04d85001a6c62addc150c0355ce95ccb69f93dd6180df4c11a2e
Sep 16 14:12:39.812778 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.812747    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:12:39.812887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.812824    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:12:39.812887 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:39.812865    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:12:40.160631 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:40.160597    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" event={"ID":"4b1b7584-889d-432f-bb61-b1d3b9ab560b","Type":"ContainerStarted","Data":"74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14"}
Sep 16 14:12:40.160774 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:40.160647    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" event={"ID":"4b1b7584-889d-432f-bb61-b1d3b9ab560b","Type":"ContainerStarted","Data":"1bb87950a9dd04d85001a6c62addc150c0355ce95ccb69f93dd6180df4c11a2e"}
Sep 16 14:12:40.185490 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:40.185434    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" podStartSLOduration=1.185416062 podStartE2EDuration="1.185416062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:40.183238245 +0000 UTC m=+1782.810719632" watchObservedRunningTime="2026-09-16 14:12:40.185416062 +0000 UTC m=+1782.812897440"
Sep 16 14:12:40.533927 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:40.533841    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:40.539302 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:40.539276    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:41.166043 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:41.166008    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:41.167033 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:41.167012    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:12:46.348312 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.348272    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"]
Sep 16 14:12:46.352073 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.352048    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:46.355967 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.355940    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-29812a-dockercfg-zgcrv\""
Sep 16 14:12:46.356093 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.355977    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-29812a\""
Sep 16 14:12:46.364294 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.364269    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"]
Sep 16 14:12:46.416542 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.416509    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebdde0d-996f-47d4-954f-e5673fb08b4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j\" (UID: \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:46.416673 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.416624    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxrff\" (UniqueName: \"kubernetes.io/projected/7ebdde0d-996f-47d4-954f-e5673fb08b4f-kube-api-access-mxrff\") pod \"payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j\" (UID: \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:46.517889 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.517859    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxrff\" (UniqueName: \"kubernetes.io/projected/7ebdde0d-996f-47d4-954f-e5673fb08b4f-kube-api-access-mxrff\") pod \"payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j\" (UID: \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:46.518071 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.517895    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebdde0d-996f-47d4-954f-e5673fb08b4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j\" (UID: \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:46.518597 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.518579    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebdde0d-996f-47d4-954f-e5673fb08b4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j\" (UID: \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:46.527303 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.527285    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxrff\" (UniqueName: \"kubernetes.io/projected/7ebdde0d-996f-47d4-954f-e5673fb08b4f-kube-api-access-mxrff\") pod \"payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j\" (UID: \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:46.664061 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.664027    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:46.949827 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:46.949803    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"]
Sep 16 14:12:46.951766 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:12:46.951738    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebdde0d_996f_47d4_954f_e5673fb08b4f.slice/crio-6e8bb38dbc250fe50207ff68e45a1bf2a66a2ec9f81836a26651ec3747553ff3 WatchSource:0}: Error finding container 6e8bb38dbc250fe50207ff68e45a1bf2a66a2ec9f81836a26651ec3747553ff3: Status 404 returned error can't find the container with id 6e8bb38dbc250fe50207ff68e45a1bf2a66a2ec9f81836a26651ec3747553ff3
Sep 16 14:12:47.200133 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:47.200036    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j" event={"ID":"7ebdde0d-996f-47d4-954f-e5673fb08b4f","Type":"ContainerStarted","Data":"0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b"}
Sep 16 14:12:47.200302 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:47.200221    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:12:47.200302 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:47.200275    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j" event={"ID":"7ebdde0d-996f-47d4-954f-e5673fb08b4f","Type":"ContainerStarted","Data":"6e8bb38dbc250fe50207ff68e45a1bf2a66a2ec9f81836a26651ec3747553ff3"}
Sep 16 14:12:47.239068 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:47.239009    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j" podStartSLOduration=1.238994346 podStartE2EDuration="1.238994346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:47.237109903 +0000 UTC m=+1789.864591280" watchObservedRunningTime="2026-09-16 14:12:47.238994346 +0000 UTC m=+1789.866475719"
Sep 16 14:12:57.955989 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:57.955960    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 14:12:57.957980 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:57.957959    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 14:12:58.173742 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:58.173716    2548 scope.go:117] "RemoveContainer" containerID="4677a6212db916c8fee4a20b3cdddbf0fb2cb19cb29bab13b64abb8bb919a21e"
Sep 16 14:12:58.208077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:12:58.208025    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:13:01.276636 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:01.276604    2548 generic.go:356] "Generic (PLEG): container finished" podID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerID="74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14" exitCode=0
Sep 16 14:13:01.277077 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:01.276684    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" event={"ID":"4b1b7584-889d-432f-bb61-b1d3b9ab560b","Type":"ContainerDied","Data":"74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14"}
Sep 16 14:13:01.277203 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:01.277186    2548 scope.go:117] "RemoveContainer" containerID="74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14"
Sep 16 14:13:01.278320 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:01.278294    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:13:01.278397 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:01.278358    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:13:01.278397 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:01.278389    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 14:13:02.287409 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:02.287363    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" event={"ID":"4b1b7584-889d-432f-bb61-b1d3b9ab560b","Type":"ContainerStarted","Data":"d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f"}
Sep 16 14:13:02.534189 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:02.534146    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:13:02.534462 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:02.534440    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 16 14:13:02.534512 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:02.534488    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 14:13:03.534275 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:03.534228    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 16 14:13:03.534644 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:03.534324    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 14:13:04.534575 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:04.534536    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 16 14:13:04.534961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:04.534608    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 14:13:05.534109 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:05.534072    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 16 14:13:05.534321 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:05.534135    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 14:13:06.534897 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:06.534858    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 16 14:13:06.535384 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:06.534923    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 14:13:07.538957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:07.538924    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:13:07.539412 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:07.539218    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:13:07.539932 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:07.539911    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:13:24.077137 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.077102    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"]
Sep 16 14:13:24.077566 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.077408    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j" podUID="7ebdde0d-996f-47d4-954f-e5673fb08b4f" containerName="payload-processing" containerID="cri-o://0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b" gracePeriod=30
Sep 16 14:13:24.336759 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.336707    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:13:24.397948 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.397916    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebdde0d-996f-47d4-954f-e5673fb08b4f-plugins-config-volume\") pod \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\" (UID: \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\") "
Sep 16 14:13:24.398094 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.397964    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxrff\" (UniqueName: \"kubernetes.io/projected/7ebdde0d-996f-47d4-954f-e5673fb08b4f-kube-api-access-mxrff\") pod \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\" (UID: \"7ebdde0d-996f-47d4-954f-e5673fb08b4f\") "
Sep 16 14:13:24.398385 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.398360    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebdde0d-996f-47d4-954f-e5673fb08b4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ebdde0d-996f-47d4-954f-e5673fb08b4f" (UID: "7ebdde0d-996f-47d4-954f-e5673fb08b4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:13:24.401806 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.401770    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebdde0d-996f-47d4-954f-e5673fb08b4f-kube-api-access-mxrff" (OuterVolumeSpecName: "kube-api-access-mxrff") pod "7ebdde0d-996f-47d4-954f-e5673fb08b4f" (UID: "7ebdde0d-996f-47d4-954f-e5673fb08b4f"). InnerVolumeSpecName "kube-api-access-mxrff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:13:24.408723 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.408692    2548 generic.go:356] "Generic (PLEG): container finished" podID="7ebdde0d-996f-47d4-954f-e5673fb08b4f" containerID="0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b" exitCode=0
Sep 16 14:13:24.408865 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.408745    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"
Sep 16 14:13:24.408865 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.408816    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j" event={"ID":"7ebdde0d-996f-47d4-954f-e5673fb08b4f","Type":"ContainerDied","Data":"0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b"}
Sep 16 14:13:24.408993 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.408865    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j" event={"ID":"7ebdde0d-996f-47d4-954f-e5673fb08b4f","Type":"ContainerDied","Data":"6e8bb38dbc250fe50207ff68e45a1bf2a66a2ec9f81836a26651ec3747553ff3"}
Sep 16 14:13:24.408993 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.408895    2548 scope.go:117] "RemoveContainer" containerID="0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b"
Sep 16 14:13:24.426514 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.426496    2548 scope.go:117] "RemoveContainer" containerID="0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b"
Sep 16 14:13:24.427056 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:13:24.426984    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b\": container with ID starting with 0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b not found: ID does not exist" containerID="0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b"
Sep 16 14:13:24.427056 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.427015    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b"} err="failed to get container status \"0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b\": rpc error: code = NotFound desc = could not find container \"0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b\": container with ID starting with 0b235654b257c360a49776beb30e0696b98362cbcf8600bbe7912a9926af6f9b not found: ID does not exist"
Sep 16 14:13:24.438410 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.438389    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"]
Sep 16 14:13:24.440342 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.440320    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29812a-5678cd67df-dfh8j"]
Sep 16 14:13:24.499322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.499297    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ebdde0d-996f-47d4-954f-e5673fb08b4f-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:24.499322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:24.499321    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxrff\" (UniqueName: \"kubernetes.io/projected/7ebdde0d-996f-47d4-954f-e5673fb08b4f-kube-api-access-mxrff\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:25.970042 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:25.970005    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ebdde0d-996f-47d4-954f-e5673fb08b4f" path="/var/lib/kubelet/pods/7ebdde0d-996f-47d4-954f-e5673fb08b4f/volumes"
Sep 16 14:13:31.480475 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:31.480436    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"]
Sep 16 14:13:31.480961 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:31.480912    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy" containerID="cri-o://d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f" gracePeriod=30
Sep 16 14:13:36.724774 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.724749    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:13:36.814357 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814290    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-token\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814357 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814331    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-socket\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814518 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814450    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-data\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814518 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814505    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-podinfo\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814606 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814533    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-certs\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814606 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814577    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-kube-api-access-zswm2\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814712 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814608    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istiod-ca-cert\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814712 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814617    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:13:36.814712 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814678    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-envoy\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814876 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814715    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-credential-socket\") pod \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\" (UID: \"4b1b7584-889d-432f-bb61-b1d3b9ab560b\") "
Sep 16 14:13:36.814876 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.814723    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:13:36.815050 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.815010    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:13:36.815050 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.815037    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:13:36.815301 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.815074    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:36.815301 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.815093    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:36.815301 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.815155    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:13:36.816525 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.816501    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:13:36.816697 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.816681    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:13:36.816760 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.816702    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:13:36.816800 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.816779    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-kube-api-access-zswm2" (OuterVolumeSpecName: "kube-api-access-zswm2") pod "4b1b7584-889d-432f-bb61-b1d3b9ab560b" (UID: "4b1b7584-889d-432f-bb61-b1d3b9ab560b"). InnerVolumeSpecName "kube-api-access-zswm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:13:36.916539 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.916511    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:36.916539 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.916536    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-kube-api-access-zswm2\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:36.916539 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.916546    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:36.916762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.916554    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:36.916762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.916562    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1b7584-889d-432f-bb61-b1d3b9ab560b-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:36.916762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.916570    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:36.916762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:36.916578    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1b7584-889d-432f-bb61-b1d3b9ab560b-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:37.483767 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.483731    2548 generic.go:356] "Generic (PLEG): container finished" podID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerID="d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f" exitCode=0
Sep 16 14:13:37.483957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.483777    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"
Sep 16 14:13:37.483957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.483797    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" event={"ID":"4b1b7584-889d-432f-bb61-b1d3b9ab560b","Type":"ContainerDied","Data":"d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f"}
Sep 16 14:13:37.483957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.483840    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9" event={"ID":"4b1b7584-889d-432f-bb61-b1d3b9ab560b","Type":"ContainerDied","Data":"1bb87950a9dd04d85001a6c62addc150c0355ce95ccb69f93dd6180df4c11a2e"}
Sep 16 14:13:37.483957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.483861    2548 scope.go:117] "RemoveContainer" containerID="d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f"
Sep 16 14:13:37.494379 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.494363    2548 scope.go:117] "RemoveContainer" containerID="74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14"
Sep 16 14:13:37.502066 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.502048    2548 scope.go:117] "RemoveContainer" containerID="d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f"
Sep 16 14:13:37.502297 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:13:37.502278    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f\": container with ID starting with d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f not found: ID does not exist" containerID="d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f"
Sep 16 14:13:37.502357 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.502305    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f"} err="failed to get container status \"d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f\": rpc error: code = NotFound desc = could not find container \"d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f\": container with ID starting with d92bb89a1a377cdd352ece2943e003faf6022da9f4a216881368f62b21993e8f not found: ID does not exist"
Sep 16 14:13:37.502357 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.502323    2548 scope.go:117] "RemoveContainer" containerID="74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14"
Sep 16 14:13:37.502607 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:13:37.502589    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14\": container with ID starting with 74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14 not found: ID does not exist" containerID="74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14"
Sep 16 14:13:37.502664 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.502610    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14"} err="failed to get container status \"74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14\": rpc error: code = NotFound desc = could not find container \"74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14\": container with ID starting with 74fe03a3a36147226c6bbc1c98aba31d627b1701e2eea64c59621c1b62e19c14 not found: ID does not exist"
Sep 16 14:13:37.508868 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.508849    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"]
Sep 16 14:13:37.513654 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.513633    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29812a-openshift-default-74c8ffb666-qz2g9"]
Sep 16 14:13:37.970538 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:37.970509    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" path="/var/lib/kubelet/pods/4b1b7584-889d-432f-bb61-b1d3b9ab560b/volumes"
Sep 16 14:13:50.745488 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:50.745453    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"]
Sep 16 14:13:50.745980 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:50.745670    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd" podUID="d87bc362-49d2-4131-ae29-07abab55883d" containerName="payload-processing" containerID="cri-o://6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86" gracePeriod=30
Sep 16 14:13:51.012517 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.012491    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:13:51.049509 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.049484    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bvcn\" (UniqueName: \"kubernetes.io/projected/d87bc362-49d2-4131-ae29-07abab55883d-kube-api-access-4bvcn\") pod \"d87bc362-49d2-4131-ae29-07abab55883d\" (UID: \"d87bc362-49d2-4131-ae29-07abab55883d\") "
Sep 16 14:13:51.049714 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.049550    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87bc362-49d2-4131-ae29-07abab55883d-plugins-config-volume\") pod \"d87bc362-49d2-4131-ae29-07abab55883d\" (UID: \"d87bc362-49d2-4131-ae29-07abab55883d\") "
Sep 16 14:13:51.049942 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.049915    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87bc362-49d2-4131-ae29-07abab55883d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d87bc362-49d2-4131-ae29-07abab55883d" (UID: "d87bc362-49d2-4131-ae29-07abab55883d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:13:51.051703 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.051668    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87bc362-49d2-4131-ae29-07abab55883d-kube-api-access-4bvcn" (OuterVolumeSpecName: "kube-api-access-4bvcn") pod "d87bc362-49d2-4131-ae29-07abab55883d" (UID: "d87bc362-49d2-4131-ae29-07abab55883d"). InnerVolumeSpecName "kube-api-access-4bvcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:13:51.150657 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.150626    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87bc362-49d2-4131-ae29-07abab55883d-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:51.150657 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.150653    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bvcn\" (UniqueName: \"kubernetes.io/projected/d87bc362-49d2-4131-ae29-07abab55883d-kube-api-access-4bvcn\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:13:51.553603 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.553569    2548 generic.go:356] "Generic (PLEG): container finished" podID="d87bc362-49d2-4131-ae29-07abab55883d" containerID="6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86" exitCode=0
Sep 16 14:13:51.553784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.553614    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"
Sep 16 14:13:51.553784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.553657    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd" event={"ID":"d87bc362-49d2-4131-ae29-07abab55883d","Type":"ContainerDied","Data":"6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86"}
Sep 16 14:13:51.553784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.553707    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd" event={"ID":"d87bc362-49d2-4131-ae29-07abab55883d","Type":"ContainerDied","Data":"35428dc3cc84252f2ae4516a77fcce31226c3b4f8e3ade49ac2e77fc6fdaf286"}
Sep 16 14:13:51.553784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.553729    2548 scope.go:117] "RemoveContainer" containerID="6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86"
Sep 16 14:13:51.569227 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.569207    2548 scope.go:117] "RemoveContainer" containerID="6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86"
Sep 16 14:13:51.569517 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:13:51.569499    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86\": container with ID starting with 6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86 not found: ID does not exist" containerID="6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86"
Sep 16 14:13:51.569565 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.569525    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86"} err="failed to get container status \"6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86\": rpc error: code = NotFound desc = could not find container \"6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86\": container with ID starting with 6bd3ca210529b68033cc2dad650b6d95abd55c6fc96f7dc145b923ba58f80f86 not found: ID does not exist"
Sep 16 14:13:51.581785 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.581756    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"]
Sep 16 14:13:51.588810 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.588790    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cabdc2-595767c6ff-whvjd"]
Sep 16 14:13:51.970443 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:51.970405    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d87bc362-49d2-4131-ae29-07abab55883d" path="/var/lib/kubelet/pods/d87bc362-49d2-4131-ae29-07abab55883d/volumes"
Sep 16 14:13:58.207582 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:13:58.207553    2548 scope.go:117] "RemoveContainer" containerID="47806588c33fc3ebf8ba3acdf1b76952a4445a70bed77321ed5f5e3ae8ba7f69"
Sep 16 14:14:46.798714 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.798681    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"]
Sep 16 14:14:46.799112 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799049    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy"
Sep 16 14:14:46.799112 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799060    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy"
Sep 16 14:14:46.799112 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799068    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebdde0d-996f-47d4-954f-e5673fb08b4f" containerName="payload-processing"
Sep 16 14:14:46.799112 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799075    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebdde0d-996f-47d4-954f-e5673fb08b4f" containerName="payload-processing"
Sep 16 14:14:46.799112 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799092    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d87bc362-49d2-4131-ae29-07abab55883d" containerName="payload-processing"
Sep 16 14:14:46.799112 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799098    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d87bc362-49d2-4131-ae29-07abab55883d" containerName="payload-processing"
Sep 16 14:14:46.799358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799160    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d87bc362-49d2-4131-ae29-07abab55883d" containerName="payload-processing"
Sep 16 14:14:46.799358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799168    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy"
Sep 16 14:14:46.799358 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.799177    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebdde0d-996f-47d4-954f-e5673fb08b4f" containerName="payload-processing"
Sep 16 14:14:46.802118 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.802102    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"
Sep 16 14:14:46.805280 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.805260    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:14:46.805384 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.805356    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:14:46.812214 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.812194    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"]
Sep 16 14:14:46.927021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:46.926990    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b884c9-b27e-45b6-94c2-70d38c031c8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk\" (UID: \"76b884c9-b27e-45b6-94c2-70d38c031c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"
Sep 16 14:14:47.027933 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:47.027901    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b884c9-b27e-45b6-94c2-70d38c031c8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk\" (UID: \"76b884c9-b27e-45b6-94c2-70d38c031c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"
Sep 16 14:14:47.028403 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:47.028385    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b884c9-b27e-45b6-94c2-70d38c031c8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk\" (UID: \"76b884c9-b27e-45b6-94c2-70d38c031c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"
Sep 16 14:14:47.113891 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:47.113809    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"
Sep 16 14:14:47.394739 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:47.394706    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"]
Sep 16 14:14:47.396431 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:14:47.396400    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b884c9_b27e_45b6_94c2_70d38c031c8d.slice/crio-11c06a51eb3d98fc3e505c24eea3ec99bb35d9a8867209d847e0a85afb3ec234 WatchSource:0}: Error finding container 11c06a51eb3d98fc3e505c24eea3ec99bb35d9a8867209d847e0a85afb3ec234: Status 404 returned error can't find the container with id 11c06a51eb3d98fc3e505c24eea3ec99bb35d9a8867209d847e0a85afb3ec234
Sep 16 14:14:47.833400 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:47.833369    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk" event={"ID":"76b884c9-b27e-45b6-94c2-70d38c031c8d","Type":"ContainerStarted","Data":"2707cc104edd653a5c09dbb920e691b436730f914850239a0e1d07d79fa2125d"}
Sep 16 14:14:47.833775 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:47.833419    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk" event={"ID":"76b884c9-b27e-45b6-94c2-70d38c031c8d","Type":"ContainerStarted","Data":"11c06a51eb3d98fc3e505c24eea3ec99bb35d9a8867209d847e0a85afb3ec234"}
Sep 16 14:14:47.856426 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:47.856382    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk" podStartSLOduration=1.8563677969999999 podStartE2EDuration="1.856367797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:47.855778276 +0000 UTC m=+1910.483259651" watchObservedRunningTime="2026-09-16 14:14:47.856367797 +0000 UTC m=+1910.483849172"
Sep 16 14:14:48.839288 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:48.839231    2548 generic.go:356] "Generic (PLEG): container finished" podID="76b884c9-b27e-45b6-94c2-70d38c031c8d" containerID="2707cc104edd653a5c09dbb920e691b436730f914850239a0e1d07d79fa2125d" exitCode=0
Sep 16 14:14:48.839683 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:48.839318    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk" event={"ID":"76b884c9-b27e-45b6-94c2-70d38c031c8d","Type":"ContainerDied","Data":"2707cc104edd653a5c09dbb920e691b436730f914850239a0e1d07d79fa2125d"}
Sep 16 14:14:49.970283 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:49.970238    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"
Sep 16 14:14:50.057841 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:50.057815    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b884c9-b27e-45b6-94c2-70d38c031c8d-maas-api-service-ca\") pod \"76b884c9-b27e-45b6-94c2-70d38c031c8d\" (UID: \"76b884c9-b27e-45b6-94c2-70d38c031c8d\") "
Sep 16 14:14:50.058155 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:50.058133    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b884c9-b27e-45b6-94c2-70d38c031c8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76b884c9-b27e-45b6-94c2-70d38c031c8d" (UID: "76b884c9-b27e-45b6-94c2-70d38c031c8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:14:50.159242 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:50.159210    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76b884c9-b27e-45b6-94c2-70d38c031c8d-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:14:50.850009 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:50.849979    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"
Sep 16 14:14:50.850175 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:50.850008    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk" event={"ID":"76b884c9-b27e-45b6-94c2-70d38c031c8d","Type":"ContainerDied","Data":"11c06a51eb3d98fc3e505c24eea3ec99bb35d9a8867209d847e0a85afb3ec234"}
Sep 16 14:14:50.850175 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:50.850056    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c06a51eb3d98fc3e505c24eea3ec99bb35d9a8867209d847e0a85afb3ec234"
Sep 16 14:14:57.561475 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.561439    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"]
Sep 16 14:14:57.561957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.561675    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff" podUID="04fbd135-6e63-43d1-96c0-686bded782cd" containerName="payload-processing" containerID="cri-o://2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2" gracePeriod=30
Sep 16 14:14:57.758817 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.758756    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"]
Sep 16 14:14:57.760501 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.760472    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9c8daa-9cxnk"]
Sep 16 14:14:57.812834 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.812771    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:14:57.897756 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.897718    2548 generic.go:356] "Generic (PLEG): container finished" podID="04fbd135-6e63-43d1-96c0-686bded782cd" containerID="2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2" exitCode=0
Sep 16 14:14:57.897914 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.897769    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"
Sep 16 14:14:57.897914 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.897804    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff" event={"ID":"04fbd135-6e63-43d1-96c0-686bded782cd","Type":"ContainerDied","Data":"2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2"}
Sep 16 14:14:57.897914 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.897888    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff" event={"ID":"04fbd135-6e63-43d1-96c0-686bded782cd","Type":"ContainerDied","Data":"2b54e007a78ae786d7ebcabc14d34d161220d3a121808522e7dfc51ad740edb1"}
Sep 16 14:14:57.898074 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.897916    2548 scope.go:117] "RemoveContainer" containerID="2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2"
Sep 16 14:14:57.908837 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.908781    2548 scope.go:117] "RemoveContainer" containerID="2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2"
Sep 16 14:14:57.909214 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:14:57.909185    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2\": container with ID starting with 2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2 not found: ID does not exist" containerID="2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2"
Sep 16 14:14:57.909511 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.909226    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2"} err="failed to get container status \"2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2\": rpc error: code = NotFound desc = could not find container \"2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2\": container with ID starting with 2f08f1547a87a35da9a8cd40518757cf3c63842020db0fedfff02e893bbe6af2 not found: ID does not exist"
Sep 16 14:14:57.928176 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.928153    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s447j\" (UniqueName: \"kubernetes.io/projected/04fbd135-6e63-43d1-96c0-686bded782cd-kube-api-access-s447j\") pod \"04fbd135-6e63-43d1-96c0-686bded782cd\" (UID: \"04fbd135-6e63-43d1-96c0-686bded782cd\") "
Sep 16 14:14:57.928302 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.928198    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fbd135-6e63-43d1-96c0-686bded782cd-plugins-config-volume\") pod \"04fbd135-6e63-43d1-96c0-686bded782cd\" (UID: \"04fbd135-6e63-43d1-96c0-686bded782cd\") "
Sep 16 14:14:57.928585 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.928564    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fbd135-6e63-43d1-96c0-686bded782cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04fbd135-6e63-43d1-96c0-686bded782cd" (UID: "04fbd135-6e63-43d1-96c0-686bded782cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:14:57.930100 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.930082    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fbd135-6e63-43d1-96c0-686bded782cd-kube-api-access-s447j" (OuterVolumeSpecName: "kube-api-access-s447j") pod "04fbd135-6e63-43d1-96c0-686bded782cd" (UID: "04fbd135-6e63-43d1-96c0-686bded782cd"). InnerVolumeSpecName "kube-api-access-s447j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:14:57.969738 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:57.969713    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b884c9-b27e-45b6-94c2-70d38c031c8d" path="/var/lib/kubelet/pods/76b884c9-b27e-45b6-94c2-70d38c031c8d/volumes"
Sep 16 14:14:58.029779 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:58.029757    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fbd135-6e63-43d1-96c0-686bded782cd-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:14:58.029779 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:58.029779    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s447j\" (UniqueName: \"kubernetes.io/projected/04fbd135-6e63-43d1-96c0-686bded782cd-kube-api-access-s447j\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:14:58.219031 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:58.219001    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"]
Sep 16 14:14:58.223816 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:58.223786    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c8daa-74497ff99-jsvff"]
Sep 16 14:14:58.236422 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:58.236393    2548 scope.go:117] "RemoveContainer" containerID="381c1d3aa139b957ba0a887a2823343fb07a2cbbe2d317480e3cd11bddf8cad6"
Sep 16 14:14:59.970012 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:14:59.969969    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04fbd135-6e63-43d1-96c0-686bded782cd" path="/var/lib/kubelet/pods/04fbd135-6e63-43d1-96c0-686bded782cd/volumes"
Sep 16 14:15:00.136094 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136066    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx"]
Sep 16 14:15:00.136451 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136438    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fbd135-6e63-43d1-96c0-686bded782cd" containerName="payload-processing"
Sep 16 14:15:00.136500 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136454    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fbd135-6e63-43d1-96c0-686bded782cd" containerName="payload-processing"
Sep 16 14:15:00.136500 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136469    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b884c9-b27e-45b6-94c2-70d38c031c8d" containerName="revoke-keys"
Sep 16 14:15:00.136500 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136474    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b884c9-b27e-45b6-94c2-70d38c031c8d" containerName="revoke-keys"
Sep 16 14:15:00.136500 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136488    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy"
Sep 16 14:15:00.136500 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136494    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy"
Sep 16 14:15:00.136658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136553    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b884c9-b27e-45b6-94c2-70d38c031c8d" containerName="revoke-keys"
Sep 16 14:15:00.136658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136561    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1b7584-889d-432f-bb61-b1d3b9ab560b" containerName="istio-proxy"
Sep 16 14:15:00.136658 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.136569    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="04fbd135-6e63-43d1-96c0-686bded782cd" containerName="payload-processing"
Sep 16 14:15:00.141023 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.141003    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx"
Sep 16 14:15:00.143505 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.143485    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:15:00.150779 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.150745    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"]
Sep 16 14:15:00.151822 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.151803    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx"
Sep 16 14:15:00.155038 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.155020    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx"]
Sep 16 14:15:00.155132 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.155121    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"
Sep 16 14:15:00.162726 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.162705    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"]
Sep 16 14:15:00.182979 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.182956    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"
Sep 16 14:15:00.447670 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.447645    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx"]
Sep 16 14:15:00.449863 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:15:00.449831    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4da1087_746d_4762_bb13_f85a7bc1af4a.slice/crio-671784790c6854da62eff3487a866cef7772470a930cc2a868621770787dbdfe WatchSource:0}: Error finding container 671784790c6854da62eff3487a866cef7772470a930cc2a868621770787dbdfe: Status 404 returned error can't find the container with id 671784790c6854da62eff3487a866cef7772470a930cc2a868621770787dbdfe
Sep 16 14:15:00.470951 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.470923    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"]
Sep 16 14:15:00.479377 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:15:00.479351    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f676414_f3c5_4662_9ae9_5414116c875a.slice/crio-a41baedf5148400ac26016bb05ab6760fd11225093642b510e5f7266ca639951 WatchSource:0}: Error finding container a41baedf5148400ac26016bb05ab6760fd11225093642b510e5f7266ca639951: Status 404 returned error can't find the container with id a41baedf5148400ac26016bb05ab6760fd11225093642b510e5f7266ca639951
Sep 16 14:15:00.916557 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.916427    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx" event={"ID":"d4da1087-746d-4762-bb13-f85a7bc1af4a","Type":"ContainerStarted","Data":"beb5d1e81d24bf61756de153b9d2409798f8aa95fa56253d7621b79308b29d20"}
Sep 16 14:15:00.916557 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.916512    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx" event={"ID":"d4da1087-746d-4762-bb13-f85a7bc1af4a","Type":"ContainerStarted","Data":"671784790c6854da62eff3487a866cef7772470a930cc2a868621770787dbdfe"}
Sep 16 14:15:00.918463 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.918432    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv" event={"ID":"9f676414-f3c5-4662-9ae9-5414116c875a","Type":"ContainerStarted","Data":"f578226fd8f7a663bf9631f83f8557076b4729d669643648a21c2906bb8202ed"}
Sep 16 14:15:00.918585 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.918484    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv" event={"ID":"9f676414-f3c5-4662-9ae9-5414116c875a","Type":"ContainerStarted","Data":"a41baedf5148400ac26016bb05ab6760fd11225093642b510e5f7266ca639951"}
Sep 16 14:15:00.934936 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.934892    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx" podStartSLOduration=0.934878924 podStartE2EDuration="934.878924ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:00.933077811 +0000 UTC m=+1923.560559198" watchObservedRunningTime="2026-09-16 14:15:00.934878924 +0000 UTC m=+1923.562360331"
Sep 16 14:15:00.951588 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:00.951547    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv" podStartSLOduration=0.951532986 podStartE2EDuration="951.532986ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:00.950634615 +0000 UTC m=+1923.578115980" watchObservedRunningTime="2026-09-16 14:15:00.951532986 +0000 UTC m=+1923.579014359"
Sep 16 14:15:01.925238 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:01.925203    2548 generic.go:356] "Generic (PLEG): container finished" podID="d4da1087-746d-4762-bb13-f85a7bc1af4a" containerID="beb5d1e81d24bf61756de153b9d2409798f8aa95fa56253d7621b79308b29d20" exitCode=0
Sep 16 14:15:01.925629 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:01.925278    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx" event={"ID":"d4da1087-746d-4762-bb13-f85a7bc1af4a","Type":"ContainerDied","Data":"beb5d1e81d24bf61756de153b9d2409798f8aa95fa56253d7621b79308b29d20"}
Sep 16 14:15:01.926911 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:01.926876    2548 generic.go:356] "Generic (PLEG): container finished" podID="9f676414-f3c5-4662-9ae9-5414116c875a" containerID="f578226fd8f7a663bf9631f83f8557076b4729d669643648a21c2906bb8202ed" exitCode=0
Sep 16 14:15:01.927014 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:01.926962    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv" event={"ID":"9f676414-f3c5-4662-9ae9-5414116c875a","Type":"ContainerDied","Data":"f578226fd8f7a663bf9631f83f8557076b4729d669643648a21c2906bb8202ed"}
Sep 16 14:15:03.080739 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:03.080712    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"
Sep 16 14:15:03.084108 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:03.084090    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx"
Sep 16 14:15:03.939115 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:03.939081    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx"
Sep 16 14:15:03.939115 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:03.939096    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826135-c4ngx" event={"ID":"d4da1087-746d-4762-bb13-f85a7bc1af4a","Type":"ContainerDied","Data":"671784790c6854da62eff3487a866cef7772470a930cc2a868621770787dbdfe"}
Sep 16 14:15:03.939360 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:03.939145    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="671784790c6854da62eff3487a866cef7772470a930cc2a868621770787dbdfe"
Sep 16 14:15:03.940450 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:03.940428    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"
Sep 16 14:15:03.940574 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:03.940425    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv" event={"ID":"9f676414-f3c5-4662-9ae9-5414116c875a","Type":"ContainerDied","Data":"a41baedf5148400ac26016bb05ab6760fd11225093642b510e5f7266ca639951"}
Sep 16 14:15:03.940574 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:03.940556    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41baedf5148400ac26016bb05ab6760fd11225093642b510e5f7266ca639951"
Sep 16 14:15:05.354677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.354646    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"]
Sep 16 14:15:05.355040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.355014    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f676414-f3c5-4662-9ae9-5414116c875a" containerName="cleanup"
Sep 16 14:15:05.355040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.355026    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f676414-f3c5-4662-9ae9-5414116c875a" containerName="cleanup"
Sep 16 14:15:05.355040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.355036    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4da1087-746d-4762-bb13-f85a7bc1af4a" containerName="cleanup"
Sep 16 14:15:05.355040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.355041    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4da1087-746d-4762-bb13-f85a7bc1af4a" containerName="cleanup"
Sep 16 14:15:05.355175 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.355095    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f676414-f3c5-4662-9ae9-5414116c875a" containerName="cleanup"
Sep 16 14:15:05.355175 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.355104    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4da1087-746d-4762-bb13-f85a7bc1af4a" containerName="cleanup"
Sep 16 14:15:05.358180 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.358160    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"
Sep 16 14:15:05.360610 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.360588    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:15:05.360610 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.360607    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:15:05.366332 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.366111    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"]
Sep 16 14:15:05.494611 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.494569    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b12aff-0115-40a4-bcb8-1e8bcbfacc63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb\" (UID: \"28b12aff-0115-40a4-bcb8-1e8bcbfacc63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"
Sep 16 14:15:05.596156 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.596118    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b12aff-0115-40a4-bcb8-1e8bcbfacc63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb\" (UID: \"28b12aff-0115-40a4-bcb8-1e8bcbfacc63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"
Sep 16 14:15:05.597178 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.597140    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b12aff-0115-40a4-bcb8-1e8bcbfacc63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb\" (UID: \"28b12aff-0115-40a4-bcb8-1e8bcbfacc63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"
Sep 16 14:15:05.671043 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.671014    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"
Sep 16 14:15:05.947638 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:05.947608    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"]
Sep 16 14:15:05.948760 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:15:05.948737    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b12aff_0115_40a4_bcb8_1e8bcbfacc63.slice/crio-a32587f8f3f4ee26d58ef9b9a8dabb8fd643ff4afedc79bc4855206c81f78bbd WatchSource:0}: Error finding container a32587f8f3f4ee26d58ef9b9a8dabb8fd643ff4afedc79bc4855206c81f78bbd: Status 404 returned error can't find the container with id a32587f8f3f4ee26d58ef9b9a8dabb8fd643ff4afedc79bc4855206c81f78bbd
Sep 16 14:15:06.957231 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:06.957153    2548 generic.go:356] "Generic (PLEG): container finished" podID="28b12aff-0115-40a4-bcb8-1e8bcbfacc63" containerID="d0f918d0c0fd14bd9380d24ec9cd8fd0687749353fad4a4724e6d35975433166" exitCode=0
Sep 16 14:15:06.957231 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:06.957220    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb" event={"ID":"28b12aff-0115-40a4-bcb8-1e8bcbfacc63","Type":"ContainerDied","Data":"d0f918d0c0fd14bd9380d24ec9cd8fd0687749353fad4a4724e6d35975433166"}
Sep 16 14:15:06.957629 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:06.957303    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb" event={"ID":"28b12aff-0115-40a4-bcb8-1e8bcbfacc63","Type":"ContainerStarted","Data":"a32587f8f3f4ee26d58ef9b9a8dabb8fd643ff4afedc79bc4855206c81f78bbd"}
Sep 16 14:15:08.092659 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:08.092629    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"
Sep 16 14:15:08.218665 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:08.218589    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b12aff-0115-40a4-bcb8-1e8bcbfacc63-maas-api-service-ca\") pod \"28b12aff-0115-40a4-bcb8-1e8bcbfacc63\" (UID: \"28b12aff-0115-40a4-bcb8-1e8bcbfacc63\") "
Sep 16 14:15:08.218987 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:08.218967    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b12aff-0115-40a4-bcb8-1e8bcbfacc63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28b12aff-0115-40a4-bcb8-1e8bcbfacc63" (UID: "28b12aff-0115-40a4-bcb8-1e8bcbfacc63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:15:08.319326 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:08.319293    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b12aff-0115-40a4-bcb8-1e8bcbfacc63-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:08.967541 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:08.967512    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"
Sep 16 14:15:08.967541 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:08.967515    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb" event={"ID":"28b12aff-0115-40a4-bcb8-1e8bcbfacc63","Type":"ContainerDied","Data":"a32587f8f3f4ee26d58ef9b9a8dabb8fd643ff4afedc79bc4855206c81f78bbd"}
Sep 16 14:15:08.967764 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:08.967570    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32587f8f3f4ee26d58ef9b9a8dabb8fd643ff4afedc79bc4855206c81f78bbd"
Sep 16 14:15:16.035538 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.035494    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"]
Sep 16 14:15:16.036002 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.035694    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n" podUID="aaa6b6b8-51d3-4d64-94e6-e65cb1681e49" containerName="payload-processing" containerID="cri-o://ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1" gracePeriod=30
Sep 16 14:15:16.066284 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.066239    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"]
Sep 16 14:15:16.066524 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.066496    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd" podUID="64e0f993-3400-4864-a3f5-898c734236f7" containerName="payload-pre-processing" containerID="cri-o://87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71" gracePeriod=30
Sep 16 14:15:16.263197 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.263138    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"]
Sep 16 14:15:16.265976 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.265951    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-46224d-gwmpb"]
Sep 16 14:15:16.354369 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.354341    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:15:16.357801 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.357779    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:15:16.493080 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.493044    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-plugins-config-volume\") pod \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\" (UID: \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\") "
Sep 16 14:15:16.493234 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.493090    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8cn2\" (UniqueName: \"kubernetes.io/projected/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-kube-api-access-q8cn2\") pod \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\" (UID: \"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49\") "
Sep 16 14:15:16.493234 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.493213    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k25m5\" (UniqueName: \"kubernetes.io/projected/64e0f993-3400-4864-a3f5-898c734236f7-kube-api-access-k25m5\") pod \"64e0f993-3400-4864-a3f5-898c734236f7\" (UID: \"64e0f993-3400-4864-a3f5-898c734236f7\") "
Sep 16 14:15:16.493352 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.493243    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e0f993-3400-4864-a3f5-898c734236f7-plugins-config-volume\") pod \"64e0f993-3400-4864-a3f5-898c734236f7\" (UID: \"64e0f993-3400-4864-a3f5-898c734236f7\") "
Sep 16 14:15:16.493487 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.493459    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aaa6b6b8-51d3-4d64-94e6-e65cb1681e49" (UID: "aaa6b6b8-51d3-4d64-94e6-e65cb1681e49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:15:16.493685 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.493662    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e0f993-3400-4864-a3f5-898c734236f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64e0f993-3400-4864-a3f5-898c734236f7" (UID: "64e0f993-3400-4864-a3f5-898c734236f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:15:16.495312 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.495293    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e0f993-3400-4864-a3f5-898c734236f7-kube-api-access-k25m5" (OuterVolumeSpecName: "kube-api-access-k25m5") pod "64e0f993-3400-4864-a3f5-898c734236f7" (UID: "64e0f993-3400-4864-a3f5-898c734236f7"). InnerVolumeSpecName "kube-api-access-k25m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:15:16.495376 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.495322    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-kube-api-access-q8cn2" (OuterVolumeSpecName: "kube-api-access-q8cn2") pod "aaa6b6b8-51d3-4d64-94e6-e65cb1681e49" (UID: "aaa6b6b8-51d3-4d64-94e6-e65cb1681e49"). InnerVolumeSpecName "kube-api-access-q8cn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:15:16.594010 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.593941    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25m5\" (UniqueName: \"kubernetes.io/projected/64e0f993-3400-4864-a3f5-898c734236f7-kube-api-access-k25m5\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:16.594010 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.593966    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e0f993-3400-4864-a3f5-898c734236f7-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:16.594010 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.593976    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:16.594010 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:16.593984    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8cn2\" (UniqueName: \"kubernetes.io/projected/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49-kube-api-access-q8cn2\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:17.004406 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.004373    2548 generic.go:356] "Generic (PLEG): container finished" podID="64e0f993-3400-4864-a3f5-898c734236f7" containerID="87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71" exitCode=0
Sep 16 14:15:17.004603 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.004415    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"
Sep 16 14:15:17.004603 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.004467    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd" event={"ID":"64e0f993-3400-4864-a3f5-898c734236f7","Type":"ContainerDied","Data":"87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71"}
Sep 16 14:15:17.004603 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.004523    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd" event={"ID":"64e0f993-3400-4864-a3f5-898c734236f7","Type":"ContainerDied","Data":"9e7ddbaaf140a31f3aec1eb803da5925add2c7ca09873a41035533b12c47c0a8"}
Sep 16 14:15:17.004603 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.004551    2548 scope.go:117] "RemoveContainer" containerID="87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71"
Sep 16 14:15:17.005834 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.005815    2548 generic.go:356] "Generic (PLEG): container finished" podID="aaa6b6b8-51d3-4d64-94e6-e65cb1681e49" containerID="ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1" exitCode=0
Sep 16 14:15:17.005924 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.005844    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n" event={"ID":"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49","Type":"ContainerDied","Data":"ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1"}
Sep 16 14:15:17.005924 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.005861    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"
Sep 16 14:15:17.005924 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.005893    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n" event={"ID":"aaa6b6b8-51d3-4d64-94e6-e65cb1681e49","Type":"ContainerDied","Data":"1855b82bb9edd5a9255ba4b070eeb6d962351f05caefe9a31973c410c046de49"}
Sep 16 14:15:17.014527 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.014508    2548 scope.go:117] "RemoveContainer" containerID="87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71"
Sep 16 14:15:17.014783 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:15:17.014764    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71\": container with ID starting with 87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71 not found: ID does not exist" containerID="87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71"
Sep 16 14:15:17.014865 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.014788    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71"} err="failed to get container status \"87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71\": rpc error: code = NotFound desc = could not find container \"87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71\": container with ID starting with 87b1e6c523fccdde96a463d37bd7fc65f7e2f70a01e63112e1fc5fd58948bf71 not found: ID does not exist"
Sep 16 14:15:17.014865 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.014805    2548 scope.go:117] "RemoveContainer" containerID="ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1"
Sep 16 14:15:17.022237 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.022222    2548 scope.go:117] "RemoveContainer" containerID="ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1"
Sep 16 14:15:17.022528 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:15:17.022509    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1\": container with ID starting with ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1 not found: ID does not exist" containerID="ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1"
Sep 16 14:15:17.022580 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.022534    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1"} err="failed to get container status \"ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1\": rpc error: code = NotFound desc = could not find container \"ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1\": container with ID starting with ab9f7e21b4c93afc0686f8c4b2c5db2ab3232095b1dfce3b894adafac393a6c1 not found: ID does not exist"
Sep 16 14:15:17.056945 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.056922    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"]
Sep 16 14:15:17.059712 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.059688    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-46224d-9dd746cf-nbt2n"]
Sep 16 14:15:17.082484 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.082462    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"]
Sep 16 14:15:17.085387 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.085368    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-46224d-85f987d46-7s4vd"]
Sep 16 14:15:17.970014 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.969978    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b12aff-0115-40a4-bcb8-1e8bcbfacc63" path="/var/lib/kubelet/pods/28b12aff-0115-40a4-bcb8-1e8bcbfacc63/volumes"
Sep 16 14:15:17.970363 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.970349    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e0f993-3400-4864-a3f5-898c734236f7" path="/var/lib/kubelet/pods/64e0f993-3400-4864-a3f5-898c734236f7/volumes"
Sep 16 14:15:17.970844 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:17.970819    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa6b6b8-51d3-4d64-94e6-e65cb1681e49" path="/var/lib/kubelet/pods/aaa6b6b8-51d3-4d64-94e6-e65cb1681e49/volumes"
Sep 16 14:15:23.795289 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:23.795242    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"]
Sep 16 14:15:23.795666 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:23.795470    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" containerID="cri-o://87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8" gracePeriod=30
Sep 16 14:15:24.181283 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181227    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"]
Sep 16 14:15:24.181682 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181663    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa6b6b8-51d3-4d64-94e6-e65cb1681e49" containerName="payload-processing"
Sep 16 14:15:24.181774 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181684    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa6b6b8-51d3-4d64-94e6-e65cb1681e49" containerName="payload-processing"
Sep 16 14:15:24.181774 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181698    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e0f993-3400-4864-a3f5-898c734236f7" containerName="payload-pre-processing"
Sep 16 14:15:24.181774 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181705    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e0f993-3400-4864-a3f5-898c734236f7" containerName="payload-pre-processing"
Sep 16 14:15:24.181774 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181725    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b12aff-0115-40a4-bcb8-1e8bcbfacc63" containerName="revoke-keys"
Sep 16 14:15:24.181774 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181734    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b12aff-0115-40a4-bcb8-1e8bcbfacc63" containerName="revoke-keys"
Sep 16 14:15:24.182024 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181814    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b12aff-0115-40a4-bcb8-1e8bcbfacc63" containerName="revoke-keys"
Sep 16 14:15:24.182024 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181827    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e0f993-3400-4864-a3f5-898c734236f7" containerName="payload-pre-processing"
Sep 16 14:15:24.182024 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.181837    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa6b6b8-51d3-4d64-94e6-e65cb1681e49" containerName="payload-processing"
Sep 16 14:15:24.186559 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.186538    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"
Sep 16 14:15:24.189153 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.189133    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-whz6l\""
Sep 16 14:15:24.189269 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.189138    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:15:24.197146 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.197123    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"]
Sep 16 14:15:24.262417 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.262383    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01118ec9-d469-4cba-989e-5350f66eb121-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v\" (UID: \"01118ec9-d469-4cba-989e-5350f66eb121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"
Sep 16 14:15:24.363702 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.363657    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01118ec9-d469-4cba-989e-5350f66eb121-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v\" (UID: \"01118ec9-d469-4cba-989e-5350f66eb121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"
Sep 16 14:15:24.364300 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.364281    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01118ec9-d469-4cba-989e-5350f66eb121-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v\" (UID: \"01118ec9-d469-4cba-989e-5350f66eb121\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"
Sep 16 14:15:24.450583 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.450511    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:15:24.450698 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.450587    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:15:24.498202 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.498171    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"
Sep 16 14:15:24.776807 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:24.776638    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"]
Sep 16 14:15:24.779383 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:15:24.779356    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01118ec9_d469_4cba_989e_5350f66eb121.slice/crio-cffe930728adf7d730f28019aab5dfb21fff65a97526ad1f7e513cd7b67fa34d WatchSource:0}: Error finding container cffe930728adf7d730f28019aab5dfb21fff65a97526ad1f7e513cd7b67fa34d: Status 404 returned error can't find the container with id cffe930728adf7d730f28019aab5dfb21fff65a97526ad1f7e513cd7b67fa34d
Sep 16 14:15:25.047882 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:25.047815    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v" event={"ID":"01118ec9-d469-4cba-989e-5350f66eb121","Type":"ContainerStarted","Data":"5372fba3d272e51320fbe6afad25c09d928a7ed3180d234c0ba7008e3bed9a3a"}
Sep 16 14:15:25.047882 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:25.047862    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v" event={"ID":"01118ec9-d469-4cba-989e-5350f66eb121","Type":"ContainerStarted","Data":"cffe930728adf7d730f28019aab5dfb21fff65a97526ad1f7e513cd7b67fa34d"}
Sep 16 14:15:25.073157 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:25.073116    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v" podStartSLOduration=1.073102748 podStartE2EDuration="1.073102748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:25.071266644 +0000 UTC m=+1947.698748013" watchObservedRunningTime="2026-09-16 14:15:25.073102748 +0000 UTC m=+1947.700584121"
Sep 16 14:15:26.054345 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:26.054317    2548 generic.go:356] "Generic (PLEG): container finished" podID="01118ec9-d469-4cba-989e-5350f66eb121" containerID="5372fba3d272e51320fbe6afad25c09d928a7ed3180d234c0ba7008e3bed9a3a" exitCode=0
Sep 16 14:15:26.054772 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:26.054384    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v" event={"ID":"01118ec9-d469-4cba-989e-5350f66eb121","Type":"ContainerDied","Data":"5372fba3d272e51320fbe6afad25c09d928a7ed3180d234c0ba7008e3bed9a3a"}
Sep 16 14:15:27.187148 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:27.187123    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"
Sep 16 14:15:27.293221 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:27.293190    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01118ec9-d469-4cba-989e-5350f66eb121-maas-api-service-ca\") pod \"01118ec9-d469-4cba-989e-5350f66eb121\" (UID: \"01118ec9-d469-4cba-989e-5350f66eb121\") "
Sep 16 14:15:27.293536 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:27.293514    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01118ec9-d469-4cba-989e-5350f66eb121-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01118ec9-d469-4cba-989e-5350f66eb121" (UID: "01118ec9-d469-4cba-989e-5350f66eb121"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:15:27.393974 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:27.393943    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01118ec9-d469-4cba-989e-5350f66eb121-maas-api-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:28.064693 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:28.064669    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"
Sep 16 14:15:28.064836 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:28.064712    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v" event={"ID":"01118ec9-d469-4cba-989e-5350f66eb121","Type":"ContainerDied","Data":"cffe930728adf7d730f28019aab5dfb21fff65a97526ad1f7e513cd7b67fa34d"}
Sep 16 14:15:28.064836 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:28.064759    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cffe930728adf7d730f28019aab5dfb21fff65a97526ad1f7e513cd7b67fa34d"
Sep 16 14:15:29.038014 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.037990    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:15:29.072131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.072066    2548 generic.go:356] "Generic (PLEG): container finished" podID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerID="87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8" exitCode=0
Sep 16 14:15:29.072131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.072106    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"
Sep 16 14:15:29.072296 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.072145    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" event={"ID":"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe","Type":"ContainerDied","Data":"87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8"}
Sep 16 14:15:29.072296 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.072199    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58" event={"ID":"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe","Type":"ContainerDied","Data":"472b51c3626ddea807f46d19ccc987b7eeaa753c01a9c611249df20339f73f43"}
Sep 16 14:15:29.072296 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.072228    2548 scope.go:117] "RemoveContainer" containerID="87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8"
Sep 16 14:15:29.080826 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.080805    2548 scope.go:117] "RemoveContainer" containerID="c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345"
Sep 16 14:15:29.088560 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.088545    2548 scope.go:117] "RemoveContainer" containerID="87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8"
Sep 16 14:15:29.088797 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:15:29.088781    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8\": container with ID starting with 87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8 not found: ID does not exist" containerID="87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8"
Sep 16 14:15:29.088861 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.088804    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8"} err="failed to get container status \"87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8\": rpc error: code = NotFound desc = could not find container \"87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8\": container with ID starting with 87c8c71379753b320c4d46242db8f4232ecb134c5edae0b1ef1269721797a7d8 not found: ID does not exist"
Sep 16 14:15:29.088861 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.088818    2548 scope.go:117] "RemoveContainer" containerID="c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345"
Sep 16 14:15:29.089043 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:15:29.089024    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345\": container with ID starting with c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345 not found: ID does not exist" containerID="c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345"
Sep 16 14:15:29.089118 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.089053    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345"} err="failed to get container status \"c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345\": rpc error: code = NotFound desc = could not find container \"c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345\": container with ID starting with c1486535d06fede1113732ece61ea02910e545a1a5a943b4530cbf0386cd2345 not found: ID does not exist"
Sep 16 14:15:29.111880 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.111855    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-certs\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.111976 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.111902    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz82q\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-kube-api-access-lz82q\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.111976 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.111971    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-credential-socket\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.112091 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112020    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-data\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.112091 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112064    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-podinfo\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.112091 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112070    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:15:29.112243 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112105    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-token\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.112243 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112140    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-envoy\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.112243 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112176    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istiod-ca-cert\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.112243 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112213    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-socket\") pod \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\" (UID: \"9f11ea68-63f3-47bd-9d30-d16f3fe3fefe\") "
Sep 16 14:15:29.112468 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112291    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:15:29.112468 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112311    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:15:29.112572 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112520    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:15:29.112625 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112564    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:15:29.112625 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112579    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.112625 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112598    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.112625 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.112613    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.114131 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.114110    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:15:29.114359 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.114341    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-kube-api-access-lz82q" (OuterVolumeSpecName: "kube-api-access-lz82q") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "kube-api-access-lz82q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:15:29.114544 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.114525    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:15:29.114544 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.114535    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" (UID: "9f11ea68-63f3-47bd-9d30-d16f3fe3fefe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:15:29.213732 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.213709    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.213732 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.213732    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.213877 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.213741    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.213877 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.213749    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.213877 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.213756    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.213877 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.213765    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz82q\" (UniqueName: \"kubernetes.io/projected/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe-kube-api-access-lz82q\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:29.399274 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.399234    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"]
Sep 16 14:15:29.404307 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.404285    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-46224d-openshift-default-8499554976-zml58"]
Sep 16 14:15:29.970062 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:29.970028    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" path="/var/lib/kubelet/pods/9f11ea68-63f3-47bd-9d30-d16f3fe3fefe/volumes"
Sep 16 14:15:34.800528 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:34.800494    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"]
Sep 16 14:15:34.803214 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:34.803191    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b4884e-29826135-b6bnv"]
Sep 16 14:15:34.836554 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:34.836524    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"]
Sep 16 14:15:34.836800 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:34.836770    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7" podUID="85246908-84cf-4090-be06-728822a42556" containerName="payload-processing" containerID="cri-o://c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73" gracePeriod=30
Sep 16 14:15:35.041685 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.041654    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"]
Sep 16 14:15:35.044937 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.044912    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b4884e-tsh4v"]
Sep 16 14:15:35.095474 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.095454    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:15:35.102596 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.102570    2548 generic.go:356] "Generic (PLEG): container finished" podID="85246908-84cf-4090-be06-728822a42556" containerID="c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73" exitCode=0
Sep 16 14:15:35.102691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.102613    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"
Sep 16 14:15:35.102691 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.102658    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7" event={"ID":"85246908-84cf-4090-be06-728822a42556","Type":"ContainerDied","Data":"c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73"}
Sep 16 14:15:35.102762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.102712    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7" event={"ID":"85246908-84cf-4090-be06-728822a42556","Type":"ContainerDied","Data":"fad6d228e03a956e3d90bb8755b91fe203e2b8cd81ceb5280215305af18ee169"}
Sep 16 14:15:35.102762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.102735    2548 scope.go:117] "RemoveContainer" containerID="c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73"
Sep 16 14:15:35.110647 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.110630    2548 scope.go:117] "RemoveContainer" containerID="c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73"
Sep 16 14:15:35.110931 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:15:35.110914    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73\": container with ID starting with c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73 not found: ID does not exist" containerID="c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73"
Sep 16 14:15:35.110989 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.110941    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73"} err="failed to get container status \"c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73\": rpc error: code = NotFound desc = could not find container \"c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73\": container with ID starting with c893fc36bbaa8697039940cf9239e4bf0e1213f35b61a6dbb8fa635cd5827e73 not found: ID does not exist"
Sep 16 14:15:35.166662 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.166631    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85246908-84cf-4090-be06-728822a42556-plugins-config-volume\") pod \"85246908-84cf-4090-be06-728822a42556\" (UID: \"85246908-84cf-4090-be06-728822a42556\") "
Sep 16 14:15:35.166835 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.166728    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtdxg\" (UniqueName: \"kubernetes.io/projected/85246908-84cf-4090-be06-728822a42556-kube-api-access-gtdxg\") pod \"85246908-84cf-4090-be06-728822a42556\" (UID: \"85246908-84cf-4090-be06-728822a42556\") "
Sep 16 14:15:35.167231 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.167206    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85246908-84cf-4090-be06-728822a42556-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85246908-84cf-4090-be06-728822a42556" (UID: "85246908-84cf-4090-be06-728822a42556"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:15:35.169302 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.169275    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85246908-84cf-4090-be06-728822a42556-kube-api-access-gtdxg" (OuterVolumeSpecName: "kube-api-access-gtdxg") pod "85246908-84cf-4090-be06-728822a42556" (UID: "85246908-84cf-4090-be06-728822a42556"). InnerVolumeSpecName "kube-api-access-gtdxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:15:35.267565 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.267534    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtdxg\" (UniqueName: \"kubernetes.io/projected/85246908-84cf-4090-be06-728822a42556-kube-api-access-gtdxg\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:35.267565 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.267559    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85246908-84cf-4090-be06-728822a42556-plugins-config-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:15:35.428833 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.428799    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"]
Sep 16 14:15:35.434001 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.433978    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b4884e-847fdff54c-k9lc7"]
Sep 16 14:15:35.969326 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.969299    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01118ec9-d469-4cba-989e-5350f66eb121" path="/var/lib/kubelet/pods/01118ec9-d469-4cba-989e-5350f66eb121/volumes"
Sep 16 14:15:35.969697 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.969683    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85246908-84cf-4090-be06-728822a42556" path="/var/lib/kubelet/pods/85246908-84cf-4090-be06-728822a42556/volumes"
Sep 16 14:15:35.969930 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:15:35.969919    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f676414-f3c5-4662-9ae9-5414116c875a" path="/var/lib/kubelet/pods/9f676414-f3c5-4662-9ae9-5414116c875a/volumes"
Sep 16 14:17:57.986445 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:17:57.986411    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 14:17:57.989589 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:17:57.989567    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xhf7x_4ca528a7-467c-4807-8dd2-5019bdc77fff/ovn-acl-logging/0.log"
Sep 16 14:18:18.342716 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:18.342684    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"]
Sep 16 14:18:18.344725 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:18.342876    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67" podUID="c74a5cfb-5249-4590-a6db-478c3dff65b3" containerName="manager" containerID="cri-o://87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f" gracePeriod=10
Sep 16 14:18:19.495499 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.495475    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 14:18:19.523200 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.523167    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bqnl\" (UniqueName: \"kubernetes.io/projected/c74a5cfb-5249-4590-a6db-478c3dff65b3-kube-api-access-7bqnl\") pod \"c74a5cfb-5249-4590-a6db-478c3dff65b3\" (UID: \"c74a5cfb-5249-4590-a6db-478c3dff65b3\") "
Sep 16 14:18:19.523431 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.523257    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c74a5cfb-5249-4590-a6db-478c3dff65b3-extensions-socket-volume\") pod \"c74a5cfb-5249-4590-a6db-478c3dff65b3\" (UID: \"c74a5cfb-5249-4590-a6db-478c3dff65b3\") "
Sep 16 14:18:19.523664 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.523641    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74a5cfb-5249-4590-a6db-478c3dff65b3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c74a5cfb-5249-4590-a6db-478c3dff65b3" (UID: "c74a5cfb-5249-4590-a6db-478c3dff65b3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:18:19.525899 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.525875    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74a5cfb-5249-4590-a6db-478c3dff65b3-kube-api-access-7bqnl" (OuterVolumeSpecName: "kube-api-access-7bqnl") pod "c74a5cfb-5249-4590-a6db-478c3dff65b3" (UID: "c74a5cfb-5249-4590-a6db-478c3dff65b3"). InnerVolumeSpecName "kube-api-access-7bqnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:18:19.624450 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.624415    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bqnl\" (UniqueName: \"kubernetes.io/projected/c74a5cfb-5249-4590-a6db-478c3dff65b3-kube-api-access-7bqnl\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:18:19.624450 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.624445    2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c74a5cfb-5249-4590-a6db-478c3dff65b3-extensions-socket-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:18:19.898154 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.898121    2548 generic.go:356] "Generic (PLEG): container finished" podID="c74a5cfb-5249-4590-a6db-478c3dff65b3" containerID="87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f" exitCode=0
Sep 16 14:18:19.898340 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.898170    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"
Sep 16 14:18:19.898340 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.898170    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67" event={"ID":"c74a5cfb-5249-4590-a6db-478c3dff65b3","Type":"ContainerDied","Data":"87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f"}
Sep 16 14:18:19.898340 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.898322    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67" event={"ID":"c74a5cfb-5249-4590-a6db-478c3dff65b3","Type":"ContainerDied","Data":"cd19247ab03ab3330371f145cc057a9160bbbb6941ff817001ae1e7d14824105"}
Sep 16 14:18:19.898503 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.898350    2548 scope.go:117] "RemoveContainer" containerID="87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f"
Sep 16 14:18:19.907701 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.907683    2548 scope.go:117] "RemoveContainer" containerID="87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f"
Sep 16 14:18:19.907989 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:18:19.907970    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f\": container with ID starting with 87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f not found: ID does not exist" containerID="87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f"
Sep 16 14:18:19.908040 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.907997    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f"} err="failed to get container status \"87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f\": rpc error: code = NotFound desc = could not find container \"87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f\": container with ID starting with 87e7cff690cb364a23611495abd837e929d3ba6599f339c8fa6dcaa275f60a1f not found: ID does not exist"
Sep 16 14:18:19.925526 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.925501    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"]
Sep 16 14:18:19.938072 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.938047    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xbn67"]
Sep 16 14:18:19.968982 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:18:19.968957    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74a5cfb-5249-4590-a6db-478c3dff65b3" path="/var/lib/kubelet/pods/c74a5cfb-5249-4590-a6db-478c3dff65b3/volumes"
Sep 16 14:19:19.048841 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.048808    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"]
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049128    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74a5cfb-5249-4590-a6db-478c3dff65b3" containerName="manager"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049139    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74a5cfb-5249-4590-a6db-478c3dff65b3" containerName="manager"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049147    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85246908-84cf-4090-be06-728822a42556" containerName="payload-processing"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049153    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="85246908-84cf-4090-be06-728822a42556" containerName="payload-processing"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049168    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049174    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049180    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01118ec9-d469-4cba-989e-5350f66eb121" containerName="revoke-keys"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049185    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="01118ec9-d469-4cba-989e-5350f66eb121" containerName="revoke-keys"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049199    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy"
Sep 16 14:19:19.049212 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049203    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy"
Sep 16 14:19:19.049530 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049258    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy"
Sep 16 14:19:19.049530 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049270    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="01118ec9-d469-4cba-989e-5350f66eb121" containerName="revoke-keys"
Sep 16 14:19:19.049530 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049275    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="85246908-84cf-4090-be06-728822a42556" containerName="payload-processing"
Sep 16 14:19:19.049530 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.049284    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74a5cfb-5249-4590-a6db-478c3dff65b3" containerName="manager"
Sep 16 14:19:19.052384 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.052367    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:19.055127 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.055112    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-khthc\""
Sep 16 14:19:19.066638 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.066614    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"]
Sep 16 14:19:19.130863 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.130836    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa55cfae-22e5-4bc8-b49c-62baed8551f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-h2cqk\" (UID: \"aa55cfae-22e5-4bc8-b49c-62baed8551f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:19.130967 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.130884    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxms7\" (UniqueName: \"kubernetes.io/projected/aa55cfae-22e5-4bc8-b49c-62baed8551f3-kube-api-access-gxms7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-h2cqk\" (UID: \"aa55cfae-22e5-4bc8-b49c-62baed8551f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:19.231421 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.231396    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxms7\" (UniqueName: \"kubernetes.io/projected/aa55cfae-22e5-4bc8-b49c-62baed8551f3-kube-api-access-gxms7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-h2cqk\" (UID: \"aa55cfae-22e5-4bc8-b49c-62baed8551f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:19.231533 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.231466    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa55cfae-22e5-4bc8-b49c-62baed8551f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-h2cqk\" (UID: \"aa55cfae-22e5-4bc8-b49c-62baed8551f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:19.231768 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.231752    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa55cfae-22e5-4bc8-b49c-62baed8551f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-h2cqk\" (UID: \"aa55cfae-22e5-4bc8-b49c-62baed8551f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:19.243414 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.243397    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxms7\" (UniqueName: \"kubernetes.io/projected/aa55cfae-22e5-4bc8-b49c-62baed8551f3-kube-api-access-gxms7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-h2cqk\" (UID: \"aa55cfae-22e5-4bc8-b49c-62baed8551f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:19.362333 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.362281    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:19.642815 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:19.642778    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"]
Sep 16 14:19:19.644346 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:19:19.644317    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa55cfae_22e5_4bc8_b49c_62baed8551f3.slice/crio-dc6db3e0f681f8c9305544bf1d78fa78f1fd61b8eba23a964242d57530567758 WatchSource:0}: Error finding container dc6db3e0f681f8c9305544bf1d78fa78f1fd61b8eba23a964242d57530567758: Status 404 returned error can't find the container with id dc6db3e0f681f8c9305544bf1d78fa78f1fd61b8eba23a964242d57530567758
Sep 16 14:19:20.175826 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:20.175794    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk" event={"ID":"aa55cfae-22e5-4bc8-b49c-62baed8551f3","Type":"ContainerStarted","Data":"62f518127c68bfa6ca14a1d81d80eb228725898ec388c1c0b459e9217d6b7549"}
Sep 16 14:19:20.176200 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:20.175849    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk" event={"ID":"aa55cfae-22e5-4bc8-b49c-62baed8551f3","Type":"ContainerStarted","Data":"dc6db3e0f681f8c9305544bf1d78fa78f1fd61b8eba23a964242d57530567758"}
Sep 16 14:19:20.176200 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:20.175875    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:19:20.200498 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:20.200452    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk" podStartSLOduration=1.200438427 podStartE2EDuration="1.200438427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:19:20.199543247 +0000 UTC m=+2182.827024655" watchObservedRunningTime="2026-09-16 14:19:20.200438427 +0000 UTC m=+2182.827919798"
Sep 16 14:19:31.183106 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:19:31.183076    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-h2cqk"
Sep 16 14:20:31.648555 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.648465    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"]
Sep 16 14:20:31.649122 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.648844    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f11ea68-63f3-47bd-9d30-d16f3fe3fefe" containerName="istio-proxy"
Sep 16 14:20:31.651959 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.651934    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.659359 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.656635    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6fdcecf1-kserve-self-signed-certs\""
Sep 16 14:20:31.666899 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.666875    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"]
Sep 16 14:20:31.758118 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.758056    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-tmp-dir\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.758322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.758131    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69l9\" (UniqueName: \"kubernetes.io/projected/c1839120-081e-4991-b7d9-c7dfb9944e69-kube-api-access-l69l9\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.758322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.758269    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1839120-081e-4991-b7d9-c7dfb9944e69-tls-certs\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.758322 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.758305    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-dshm\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.758472 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.758343    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-model-cache\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.758472 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.758361    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-home\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.858807 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.858775    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69l9\" (UniqueName: \"kubernetes.io/projected/c1839120-081e-4991-b7d9-c7dfb9944e69-kube-api-access-l69l9\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.859029 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.858863    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1839120-081e-4991-b7d9-c7dfb9944e69-tls-certs\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.859029 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.858896    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-dshm\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.859178 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.859041    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-model-cache\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.859178 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.859091    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-home\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.859178 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.859159    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-tmp-dir\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.859451 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.859426    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-model-cache\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.859530 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.859451    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-home\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.859530 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.859515    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-tmp-dir\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.861151 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.861123    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-dshm\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.861298 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.861282    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1839120-081e-4991-b7d9-c7dfb9944e69-tls-certs\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.868689 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.868670    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69l9\" (UniqueName: \"kubernetes.io/projected/c1839120-081e-4991-b7d9-c7dfb9944e69-kube-api-access-l69l9\") pod \"e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") " pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:31.965295 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:31.965206    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:32.241281 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:32.241181    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"]
Sep 16 14:20:32.243561 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:20:32.243523    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1839120_081e_4991_b7d9_c7dfb9944e69.slice/crio-e706d0b1c835138de15db45460033df8bdfccf9bd017040ab53d86b24dc7b6f6 WatchSource:0}: Error finding container e706d0b1c835138de15db45460033df8bdfccf9bd017040ab53d86b24dc7b6f6: Status 404 returned error can't find the container with id e706d0b1c835138de15db45460033df8bdfccf9bd017040ab53d86b24dc7b6f6
Sep 16 14:20:32.523714 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:32.523627    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk" event={"ID":"c1839120-081e-4991-b7d9-c7dfb9944e69","Type":"ContainerStarted","Data":"82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2"}
Sep 16 14:20:32.523714 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:32.523669    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk" event={"ID":"c1839120-081e-4991-b7d9-c7dfb9944e69","Type":"ContainerStarted","Data":"e706d0b1c835138de15db45460033df8bdfccf9bd017040ab53d86b24dc7b6f6"}
Sep 16 14:20:32.546175 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:32.546132    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk" podStartSLOduration=1.546116355 podStartE2EDuration="1.546116355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:32.544379237 +0000 UTC m=+2255.171860611" watchObservedRunningTime="2026-09-16 14:20:32.546116355 +0000 UTC m=+2255.173597730"
Sep 16 14:20:41.969218 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:41.969186    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:41.969678 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:41.969233    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:41.977948 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:41.977926    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:42.581091 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:42.581061    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:20:58.314635 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:20:58.314605    2548 scope.go:117] "RemoveContainer" containerID="2707cc104edd653a5c09dbb920e691b436730f914850239a0e1d07d79fa2125d"
Sep 16 14:21:16.367763 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:16.367722    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"]
Sep 16 14:21:31.392331 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.392290    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk" podUID="c1839120-081e-4991-b7d9-c7dfb9944e69" containerName="main" containerID="cri-o://82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2" gracePeriod=45
Sep 16 14:21:31.424310 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:21:31.424273    2548 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1839120_081e_4991_b7d9_c7dfb9944e69.slice/crio-conmon-82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1839120_081e_4991_b7d9_c7dfb9944e69.slice/crio-82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 14:21:31.640123 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.640098    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:21:31.749078 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.748991    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1839120-081e-4991-b7d9-c7dfb9944e69-tls-certs\") pod \"c1839120-081e-4991-b7d9-c7dfb9944e69\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") "
Sep 16 14:21:31.749078 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749073    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l69l9\" (UniqueName: \"kubernetes.io/projected/c1839120-081e-4991-b7d9-c7dfb9944e69-kube-api-access-l69l9\") pod \"c1839120-081e-4991-b7d9-c7dfb9944e69\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") "
Sep 16 14:21:31.749293 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749103    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-model-cache\") pod \"c1839120-081e-4991-b7d9-c7dfb9944e69\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") "
Sep 16 14:21:31.749293 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749130    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-dshm\") pod \"c1839120-081e-4991-b7d9-c7dfb9944e69\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") "
Sep 16 14:21:31.749293 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749170    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-tmp-dir\") pod \"c1839120-081e-4991-b7d9-c7dfb9944e69\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") "
Sep 16 14:21:31.749293 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749206    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-home\") pod \"c1839120-081e-4991-b7d9-c7dfb9944e69\" (UID: \"c1839120-081e-4991-b7d9-c7dfb9944e69\") "
Sep 16 14:21:31.749460 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749411    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1839120-081e-4991-b7d9-c7dfb9944e69" (UID: "c1839120-081e-4991-b7d9-c7dfb9944e69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:21:31.749518 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749474    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1839120-081e-4991-b7d9-c7dfb9944e69" (UID: "c1839120-081e-4991-b7d9-c7dfb9944e69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:21:31.749578 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749542    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-model-cache\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:31.749656 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.749637    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-home" (OuterVolumeSpecName: "home") pod "c1839120-081e-4991-b7d9-c7dfb9944e69" (UID: "c1839120-081e-4991-b7d9-c7dfb9944e69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:21:31.751453 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.751420    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-dshm" (OuterVolumeSpecName: "dshm") pod "c1839120-081e-4991-b7d9-c7dfb9944e69" (UID: "c1839120-081e-4991-b7d9-c7dfb9944e69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:21:31.751453 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.751441    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1839120-081e-4991-b7d9-c7dfb9944e69-kube-api-access-l69l9" (OuterVolumeSpecName: "kube-api-access-l69l9") pod "c1839120-081e-4991-b7d9-c7dfb9944e69" (UID: "c1839120-081e-4991-b7d9-c7dfb9944e69"). InnerVolumeSpecName "kube-api-access-l69l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:21:31.751616 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.751480    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1839120-081e-4991-b7d9-c7dfb9944e69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1839120-081e-4991-b7d9-c7dfb9944e69" (UID: "c1839120-081e-4991-b7d9-c7dfb9944e69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:21:31.806824 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.806791    2548 generic.go:356] "Generic (PLEG): container finished" podID="c1839120-081e-4991-b7d9-c7dfb9944e69" containerID="82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2" exitCode=0
Sep 16 14:21:31.806980 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.806835    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"
Sep 16 14:21:31.806980 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.806859    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk" event={"ID":"c1839120-081e-4991-b7d9-c7dfb9944e69","Type":"ContainerDied","Data":"82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2"}
Sep 16 14:21:31.806980 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.806899    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk" event={"ID":"c1839120-081e-4991-b7d9-c7dfb9944e69","Type":"ContainerDied","Data":"e706d0b1c835138de15db45460033df8bdfccf9bd017040ab53d86b24dc7b6f6"}
Sep 16 14:21:31.806980 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.806921    2548 scope.go:117] "RemoveContainer" containerID="82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2"
Sep 16 14:21:31.815280 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.815243    2548 scope.go:117] "RemoveContainer" containerID="82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2"
Sep 16 14:21:31.815529 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:21:31.815509    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2\": container with ID starting with 82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2 not found: ID does not exist" containerID="82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2"
Sep 16 14:21:31.815581 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.815536    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2"} err="failed to get container status \"82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2\": rpc error: code = NotFound desc = could not find container \"82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2\": container with ID starting with 82eac1ab0074c5567ad874f21c61c6a19d664cbd5e98427cc4c5cabcd0579ef2 not found: ID does not exist"
Sep 16 14:21:31.846794 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.846773    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"]
Sep 16 14:21:31.850436 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.850419    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-tmp-dir\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:31.850497 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.850438    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-home\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:31.850497 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.850447    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1839120-081e-4991-b7d9-c7dfb9944e69-tls-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:31.850497 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.850455    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l69l9\" (UniqueName: \"kubernetes.io/projected/c1839120-081e-4991-b7d9-c7dfb9944e69-kube-api-access-l69l9\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:31.850497 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.850464    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1839120-081e-4991-b7d9-c7dfb9944e69-dshm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:31.858749 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.858732    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6fdcecf1-kserve-7d984f6575-zpvmk"]
Sep 16 14:21:31.969381 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:31.969341    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1839120-081e-4991-b7d9-c7dfb9944e69" path="/var/lib/kubelet/pods/c1839120-081e-4991-b7d9-c7dfb9944e69/volumes"
Sep 16 14:21:35.500908 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.500867    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"]
Sep 16 14:21:35.501427 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.501408    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1839120-081e-4991-b7d9-c7dfb9944e69" containerName="main"
Sep 16 14:21:35.501509 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.501429    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1839120-081e-4991-b7d9-c7dfb9944e69" containerName="main"
Sep 16 14:21:35.501564 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.501513    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1839120-081e-4991-b7d9-c7dfb9944e69" containerName="main"
Sep 16 14:21:35.504457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.504435    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.508432 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.508411    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 14:21:35.508665 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.508645    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kq7tn\""
Sep 16 14:21:35.508805 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.508757    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:21:35.508879 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.508837    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 14:21:35.508965 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.508909    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 14:21:35.523229 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.523208    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"]
Sep 16 14:21:35.583885 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.583853    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-openshift-service-ca\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.584016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.583888    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-metrics-certs\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.584016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.583955    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-maas-api-tls\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.584016 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.583971    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp4rn\" (UniqueName: \"kubernetes.io/projected/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-kube-api-access-hp4rn\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.684694 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.684666    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-maas-api-tls\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.684694 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.684695    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp4rn\" (UniqueName: \"kubernetes.io/projected/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-kube-api-access-hp4rn\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.684902 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.684846    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-openshift-service-ca\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.684902 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.684889    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-metrics-certs\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.685579 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.685558    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-openshift-service-ca\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.687231 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.687203    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-metrics-certs\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.687432 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.687413    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-maas-api-tls\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.698108 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.698091    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp4rn\" (UniqueName: \"kubernetes.io/projected/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-kube-api-access-hp4rn\") pod \"maas-api-557699c6fc-jk7nf\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") " pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:35.814756 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:35.814664    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:36.110132 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:36.110098    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"]
Sep 16 14:21:36.113447 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:36.113422    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:21:36.833714 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:36.833667    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf" event={"ID":"ad495fdb-ca87-4958-ae56-c5a5178ab0fb","Type":"ContainerStarted","Data":"c4df665995cc0804b5452c8ee677c045d0fcd0bd1a78b2c697731ee8351d375a"}
Sep 16 14:21:38.846347 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:38.846229    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf" event={"ID":"ad495fdb-ca87-4958-ae56-c5a5178ab0fb","Type":"ContainerStarted","Data":"8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b"}
Sep 16 14:21:38.846347 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:38.846324    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:38.867230 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:38.867183    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf" podStartSLOduration=1.5307411050000002 podStartE2EDuration="3.867169073s" totalImagesPullingTime="2.336427968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:21:38.86644285 +0000 UTC m=+2321.493924224" watchObservedRunningTime="2026-09-16 14:21:38.867169073 +0000 UTC m=+2321.494650446"
Sep 16 14:21:44.166466 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.166419    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"]
Sep 16 14:21:44.166958 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.166626    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf" podUID="ad495fdb-ca87-4958-ae56-c5a5178ab0fb" containerName="maas-api" containerID="cri-o://8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b" gracePeriod=30
Sep 16 14:21:44.172572 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.172546    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:44.265393 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.265364    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"]
Sep 16 14:21:44.268781 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.268758    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.290375 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.290342    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"]
Sep 16 14:21:44.444665 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.444639    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:44.464596 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.464573    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtm6\" (UniqueName: \"kubernetes.io/projected/b2c8efab-fd52-4615-8c72-943cf456e0ba-kube-api-access-xwtm6\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.464717 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.464635    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2c8efab-fd52-4615-8c72-943cf456e0ba-maas-api-tls\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.464717 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.464676    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2c8efab-fd52-4615-8c72-943cf456e0ba-metrics-certs\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.464717 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.464694    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c8efab-fd52-4615-8c72-943cf456e0ba-openshift-service-ca\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.565438 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565406    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-metrics-certs\") pod \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") "
Sep 16 14:21:44.565624 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565447    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp4rn\" (UniqueName: \"kubernetes.io/projected/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-kube-api-access-hp4rn\") pod \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") "
Sep 16 14:21:44.565624 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565510    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-openshift-service-ca\") pod \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") "
Sep 16 14:21:44.565624 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565534    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-maas-api-tls\") pod \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\" (UID: \"ad495fdb-ca87-4958-ae56-c5a5178ab0fb\") "
Sep 16 14:21:44.565866 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565627    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c8efab-fd52-4615-8c72-943cf456e0ba-openshift-service-ca\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.565866 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565694    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtm6\" (UniqueName: \"kubernetes.io/projected/b2c8efab-fd52-4615-8c72-943cf456e0ba-kube-api-access-xwtm6\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.565866 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565847    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2c8efab-fd52-4615-8c72-943cf456e0ba-maas-api-tls\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.566021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565898    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad495fdb-ca87-4958-ae56-c5a5178ab0fb" (UID: "ad495fdb-ca87-4958-ae56-c5a5178ab0fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:21:44.566021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565910    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2c8efab-fd52-4615-8c72-943cf456e0ba-metrics-certs\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.566021 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.565982    2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:44.566564 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.566533    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c8efab-fd52-4615-8c72-943cf456e0ba-openshift-service-ca\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.568005 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.567940    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad495fdb-ca87-4958-ae56-c5a5178ab0fb" (UID: "ad495fdb-ca87-4958-ae56-c5a5178ab0fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:21:44.568107 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.568009    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad495fdb-ca87-4958-ae56-c5a5178ab0fb" (UID: "ad495fdb-ca87-4958-ae56-c5a5178ab0fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:21:44.568107 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.568086    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-kube-api-access-hp4rn" (OuterVolumeSpecName: "kube-api-access-hp4rn") pod "ad495fdb-ca87-4958-ae56-c5a5178ab0fb" (UID: "ad495fdb-ca87-4958-ae56-c5a5178ab0fb"). InnerVolumeSpecName "kube-api-access-hp4rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:21:44.568412 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.568391    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2c8efab-fd52-4615-8c72-943cf456e0ba-metrics-certs\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.568592 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.568578    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2c8efab-fd52-4615-8c72-943cf456e0ba-maas-api-tls\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.593864 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.593842    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtm6\" (UniqueName: \"kubernetes.io/projected/b2c8efab-fd52-4615-8c72-943cf456e0ba-kube-api-access-xwtm6\") pod \"maas-api-fd9d6989-7lc58\" (UID: \"b2c8efab-fd52-4615-8c72-943cf456e0ba\") " pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.666968 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.666943    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-metrics-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:44.666968 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.666965    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp4rn\" (UniqueName: \"kubernetes.io/projected/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-kube-api-access-hp4rn\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:44.667102 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.666975    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad495fdb-ca87-4958-ae56-c5a5178ab0fb-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\""
Sep 16 14:21:44.875581 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.875552    2548 generic.go:356] "Generic (PLEG): container finished" podID="ad495fdb-ca87-4958-ae56-c5a5178ab0fb" containerID="8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b" exitCode=0
Sep 16 14:21:44.875762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.875591    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"
Sep 16 14:21:44.875762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.875634    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf" event={"ID":"ad495fdb-ca87-4958-ae56-c5a5178ab0fb","Type":"ContainerDied","Data":"8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b"}
Sep 16 14:21:44.875762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.875693    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf" event={"ID":"ad495fdb-ca87-4958-ae56-c5a5178ab0fb","Type":"ContainerDied","Data":"c4df665995cc0804b5452c8ee677c045d0fcd0bd1a78b2c697731ee8351d375a"}
Sep 16 14:21:44.875762 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.875724    2548 scope.go:117] "RemoveContainer" containerID="8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b"
Sep 16 14:21:44.879735 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.879715    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:44.884999 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.884982    2548 scope.go:117] "RemoveContainer" containerID="8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b"
Sep 16 14:21:44.885299 ip-10-0-129-226 kubenswrapper[2548]: E0916 14:21:44.885277    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b\": container with ID starting with 8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b not found: ID does not exist" containerID="8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b"
Sep 16 14:21:44.885379 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.885305    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b"} err="failed to get container status \"8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b\": rpc error: code = NotFound desc = could not find container \"8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b\": container with ID starting with 8e7ab18b3401900206ab43e09c81d43344f5986307d8b018d56116c519d2bc6b not found: ID does not exist"
Sep 16 14:21:44.908997 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.908966    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"]
Sep 16 14:21:44.911919 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:44.911885    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-557699c6fc-jk7nf"]
Sep 16 14:21:45.163935 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:45.163913    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"]
Sep 16 14:21:45.165570 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:21:45.165549    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c8efab_fd52_4615_8c72_943cf456e0ba.slice/crio-75af22b29de6eee75d1054c86f0f4667a710c29e2faaba44f584619a1f8e3eaf WatchSource:0}: Error finding container 75af22b29de6eee75d1054c86f0f4667a710c29e2faaba44f584619a1f8e3eaf: Status 404 returned error can't find the container with id 75af22b29de6eee75d1054c86f0f4667a710c29e2faaba44f584619a1f8e3eaf
Sep 16 14:21:45.889240 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:45.889206    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58" event={"ID":"b2c8efab-fd52-4615-8c72-943cf456e0ba","Type":"ContainerStarted","Data":"da242ab831de8ae2ffec35b5a59d2906c40a4773990b1b36e4d3491d2487cbc0"}
Sep 16 14:21:45.889657 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:45.889298    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58" event={"ID":"b2c8efab-fd52-4615-8c72-943cf456e0ba","Type":"ContainerStarted","Data":"75af22b29de6eee75d1054c86f0f4667a710c29e2faaba44f584619a1f8e3eaf"}
Sep 16 14:21:45.889657 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:45.889390    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:45.909598 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:45.909552    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58" podStartSLOduration=1.488139076 podStartE2EDuration="1.909537041s" totalImagesPullingTime="421.397965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:21:45.907663674 +0000 UTC m=+2328.535145049" watchObservedRunningTime="2026-09-16 14:21:45.909537041 +0000 UTC m=+2328.537018425"
Sep 16 14:21:45.970271 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:45.970221    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad495fdb-ca87-4958-ae56-c5a5178ab0fb" path="/var/lib/kubelet/pods/ad495fdb-ca87-4958-ae56-c5a5178ab0fb/volumes"
Sep 16 14:21:51.899544 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:51.899482    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fd9d6989-7lc58"
Sep 16 14:21:58.333601 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:58.333573    2548 scope.go:117] "RemoveContainer" containerID="d0f918d0c0fd14bd9380d24ec9cd8fd0687749353fad4a4724e6d35975433166"
Sep 16 14:21:58.341450 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:58.341428    2548 scope.go:117] "RemoveContainer" containerID="5372fba3d272e51320fbe6afad25c09d928a7ed3180d234c0ba7008e3bed9a3a"
Sep 16 14:21:58.348628 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:21:58.348608    2548 scope.go:117] "RemoveContainer" containerID="f578226fd8f7a663bf9631f83f8557076b4729d669643648a21c2906bb8202ed"
Sep 16 14:22:12.162033 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:12.162001    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xwcrn_94bc5768-cee6-4110-b0d1-9b5826bdb8f0/manager/0.log"
Sep 16 14:22:12.541673 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:12.541596    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-257vw_8252f0a8-eaec-4b91-8ca5-2350f5a31405/server/0.log"
Sep 16 14:22:12.908869 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:12.908842    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-h8t7z_bb874d95-6b2b-41eb-acab-aaff3c882a3e/copy-manifests/0.log"
Sep 16 14:22:12.917091 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:12.917068    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-h8t7z_bb874d95-6b2b-41eb-acab-aaff3c882a3e/manager/0.log"
Sep 16 14:22:13.372204 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:13.372172    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fd9d6989-7lc58_b2c8efab-fd52-4615-8c72-943cf456e0ba/maas-api/0.log"
Sep 16 14:22:13.598013 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:13.597985    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826120-4gz7n_8d852a22-3f3f-4b75-96b4-e44f27488976/cleanup/0.log"
Sep 16 14:22:13.715857 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:13.715776    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826135-c4ngx_d4da1087-746d-4762-bb13-f85a7bc1af4a/cleanup/0.log"
Sep 16 14:22:14.716555 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:14.716524    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p7js2_0561a9a3-4b3f-4d21-8aa6-aa76148f90d1/manager/0.log"
Sep 16 14:22:14.838002 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:14.837977    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-swdf7_4081aa12-adb5-41e0-bded-1eea778917f9/manager/0.log"
Sep 16 14:22:15.079084 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:15.079003    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-h2cqk_aa55cfae-22e5-4bc8-b49c-62baed8551f3/manager/0.log"
Sep 16 14:22:15.680547 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:15.680511    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq_ae21a4ff-f37f-43cd-876d-1351113bb384/istio-proxy/0.log"
Sep 16 14:22:16.157481 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:16.157446    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-l5hkn_1ce50390-e6e1-49fe-9e01-8e74baea14ac/istio-proxy/1.log"
Sep 16 14:22:16.274380 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:16.274350    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wzccf_01a23ec2-ae3a-4d16-95b2-8ed7ede19084/payload-pre-processing/0.log"
Sep 16 14:22:16.974315 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:16.974274    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7lqql_2f689721-b83b-4bb5-b959-997b2cdd0223/main/0.log"
Sep 16 14:22:17.348062 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:17.347964    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnkv6n_6a25fda7-a805-44e1-ad79-810b11c7a467/main/0.log"
Sep 16 14:22:26.907265 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:26.907225    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wqkrw_cdae81bc-d373-442e-90b2-a48654b28fb6/global-pull-secret-syncer/0.log"
Sep 16 14:22:27.078573 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:27.078545    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mnvnl_7cb0ef61-c61a-4d07-a6e2-9b4dc5852ef5/konnectivity-agent/0.log"
Sep 16 14:22:27.196453 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:27.196366    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-226.ec2.internal_611a46b14cbb4ffe0f7c676f67e60bdf/haproxy/0.log"
Sep 16 14:22:31.913466 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:31.913436    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p7js2_0561a9a3-4b3f-4d21-8aa6-aa76148f90d1/manager/0.log"
Sep 16 14:22:31.940187 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:31.940166    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-swdf7_4081aa12-adb5-41e0-bded-1eea778917f9/manager/0.log"
Sep 16 14:22:32.025740 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:32.025713    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-h2cqk_aa55cfae-22e5-4bc8-b49c-62baed8551f3/manager/0.log"
Sep 16 14:22:33.969533 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:33.969504    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2v8pl_7c6e42ac-e552-49bb-879d-1c37ad6cfb35/kube-state-metrics/0.log"
Sep 16 14:22:33.998999 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:33.998975    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2v8pl_7c6e42ac-e552-49bb-879d-1c37ad6cfb35/kube-rbac-proxy-main/0.log"
Sep 16 14:22:34.031102 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.031079    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2v8pl_7c6e42ac-e552-49bb-879d-1c37ad6cfb35/kube-rbac-proxy-self/0.log"
Sep 16 14:22:34.168738 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.168666    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tt2w_51b9a334-5b84-4317-a09d-06a3ece75cbe/node-exporter/0.log"
Sep 16 14:22:34.196444 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.196418    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tt2w_51b9a334-5b84-4317-a09d-06a3ece75cbe/kube-rbac-proxy/0.log"
Sep 16 14:22:34.231389 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.231369    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9tt2w_51b9a334-5b84-4317-a09d-06a3ece75cbe/init-textfile/0.log"
Sep 16 14:22:34.791047 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.791015    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1b1f6d4-1d74-4478-9e51-acd79d0ec89c/prometheus/0.log"
Sep 16 14:22:34.834678 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.834609    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1b1f6d4-1d74-4478-9e51-acd79d0ec89c/config-reloader/0.log"
Sep 16 14:22:34.890906 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.890879    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1b1f6d4-1d74-4478-9e51-acd79d0ec89c/thanos-sidecar/0.log"
Sep 16 14:22:34.927666 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.927634    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1b1f6d4-1d74-4478-9e51-acd79d0ec89c/kube-rbac-proxy-web/0.log"
Sep 16 14:22:34.955190 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:34.955167    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1b1f6d4-1d74-4478-9e51-acd79d0ec89c/kube-rbac-proxy/0.log"
Sep 16 14:22:35.000147 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:35.000122    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1b1f6d4-1d74-4478-9e51-acd79d0ec89c/kube-rbac-proxy-thanos/0.log"
Sep 16 14:22:35.040223 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:35.040200    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1b1f6d4-1d74-4478-9e51-acd79d0ec89c/init-config-reloader/0.log"
Sep 16 14:22:37.533181 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.533150    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"]
Sep 16 14:22:37.533584 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.533497    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad495fdb-ca87-4958-ae56-c5a5178ab0fb" containerName="maas-api"
Sep 16 14:22:37.533584 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.533509    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad495fdb-ca87-4958-ae56-c5a5178ab0fb" containerName="maas-api"
Sep 16 14:22:37.533584 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.533566    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad495fdb-ca87-4958-ae56-c5a5178ab0fb" containerName="maas-api"
Sep 16 14:22:37.536404 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.536380    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.542525 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.542506    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xhpr\"/\"default-dockercfg-wm7b4\""
Sep 16 14:22:37.543183 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.543164    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xhpr\"/\"openshift-service-ca.crt\""
Sep 16 14:22:37.543527 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.543510    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xhpr\"/\"kube-root-ca.crt\""
Sep 16 14:22:37.556482 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.556458    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"]
Sep 16 14:22:37.705502 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.705478    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvg7\" (UniqueName: \"kubernetes.io/projected/250b3c19-8875-4553-9c46-4f732dabf76a-kube-api-access-8rvg7\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.705628 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.705511    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-sys\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.705628 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.705537    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-podres\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.705628 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.705617    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-proc\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.705775 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.705643    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-lib-modules\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.806927 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.806866    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-lib-modules\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.806927 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.806913    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvg7\" (UniqueName: \"kubernetes.io/projected/250b3c19-8875-4553-9c46-4f732dabf76a-kube-api-access-8rvg7\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.807104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.806936    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-sys\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.807104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.806967    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-podres\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.807104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.806997    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-proc\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.807104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.807030    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-lib-modules\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.807104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.807071    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-proc\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.807104 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.807071    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-sys\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.807367 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.807113    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/250b3c19-8875-4553-9c46-4f732dabf76a-podres\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.817948 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.817924    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvg7\" (UniqueName: \"kubernetes.io/projected/250b3c19-8875-4553-9c46-4f732dabf76a-kube-api-access-8rvg7\") pod \"perf-node-gather-daemonset-d2shg\" (UID: \"250b3c19-8875-4553-9c46-4f732dabf76a\") " pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:37.845347 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:37.845324    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:38.122296 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:38.122274    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"]
Sep 16 14:22:38.123854 ip-10-0-129-226 kubenswrapper[2548]: W0916 14:22:38.123828    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod250b3c19_8875_4553_9c46_4f732dabf76a.slice/crio-8d80952dafc6dec388d4134fa2920510b0b608125cd5dfc26e858dec0d1441ff WatchSource:0}: Error finding container 8d80952dafc6dec388d4134fa2920510b0b608125cd5dfc26e858dec0d1441ff: Status 404 returned error can't find the container with id 8d80952dafc6dec388d4134fa2920510b0b608125cd5dfc26e858dec0d1441ff
Sep 16 14:22:39.122308 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:39.122273    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg" event={"ID":"250b3c19-8875-4553-9c46-4f732dabf76a","Type":"ContainerStarted","Data":"013eec14f904112d1765af179cf1d74dbabafb01c95850284490c05af95067c8"}
Sep 16 14:22:39.122677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:39.122333    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:39.122677 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:39.122357    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg" event={"ID":"250b3c19-8875-4553-9c46-4f732dabf76a","Type":"ContainerStarted","Data":"8d80952dafc6dec388d4134fa2920510b0b608125cd5dfc26e858dec0d1441ff"}
Sep 16 14:22:39.146576 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:39.146527    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg" podStartSLOduration=2.146512879 podStartE2EDuration="2.146512879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:39.144827931 +0000 UTC m=+2381.772309303" watchObservedRunningTime="2026-09-16 14:22:39.146512879 +0000 UTC m=+2381.773994252"
Sep 16 14:22:40.451916 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:40.451882    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bxzwx_ef388d95-2cfe-4771-aa31-0128b60b0723/dns/0.log"
Sep 16 14:22:40.482710 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:40.482687    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bxzwx_ef388d95-2cfe-4771-aa31-0128b60b0723/kube-rbac-proxy/0.log"
Sep 16 14:22:40.734940 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:40.734857    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xb4zs_1d932d2e-0c6f-4068-8358-6d091cc9b123/dns-node-resolver/0.log"
Sep 16 14:22:41.563445 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:41.563414    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sklmz_809e016e-5879-42bd-83c4-ea90a7d66cfd/node-ca/0.log"
Sep 16 14:22:42.665646 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:42.665614    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq_ae21a4ff-f37f-43cd-876d-1351113bb384/istio-proxy/0.log"
Sep 16 14:22:43.553594 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:43.553554    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-l5hkn_1ce50390-e6e1-49fe-9e01-8e74baea14ac/istio-proxy/0.log"
Sep 16 14:22:43.557822 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:43.557801    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-l5hkn_1ce50390-e6e1-49fe-9e01-8e74baea14ac/istio-proxy/1.log"
Sep 16 14:22:43.606031 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:43.606001    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wzccf_01a23ec2-ae3a-4d16-95b2-8ed7ede19084/payload-pre-processing/0.log"
Sep 16 14:22:44.704947 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:44.704918    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gh2dv_5cca1368-1fe3-483a-a299-13b6f79aa8ac/serve-healthcheck-canary/0.log"
Sep 16 14:22:45.134425 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:45.134400    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xhpr/perf-node-gather-daemonset-d2shg"
Sep 16 14:22:45.450457 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:45.450380    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shkw6_d1ba3d15-8156-4d5d-916f-e6538b6b91a1/kube-rbac-proxy/0.log"
Sep 16 14:22:45.473974 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:45.473949    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shkw6_d1ba3d15-8156-4d5d-916f-e6538b6b91a1/exporter/0.log"
Sep 16 14:22:45.499957 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:45.499937    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shkw6_d1ba3d15-8156-4d5d-916f-e6538b6b91a1/extractor/0.log"
Sep 16 14:22:47.665800 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:47.665765    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq_ae21a4ff-f37f-43cd-876d-1351113bb384/istio-proxy/0.log"
Sep 16 14:22:47.981680 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:47.981647    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xwcrn_94bc5768-cee6-4110-b0d1-9b5826bdb8f0/manager/0.log"
Sep 16 14:22:48.740332 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:48.740304    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-l5hkn_1ce50390-e6e1-49fe-9e01-8e74baea14ac/istio-proxy/0.log"
Sep 16 14:22:48.744788 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:48.744761    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-l5hkn_1ce50390-e6e1-49fe-9e01-8e74baea14ac/istio-proxy/1.log"
Sep 16 14:22:48.766748 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:48.766718    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-257vw_8252f0a8-eaec-4b91-8ca5-2350f5a31405/server/0.log"
Sep 16 14:22:48.775124 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:48.775103    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wzccf_01a23ec2-ae3a-4d16-95b2-8ed7ede19084/payload-pre-processing/0.log"
Sep 16 14:22:49.041059 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:49.040983    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-h8t7z_bb874d95-6b2b-41eb-acab-aaff3c882a3e/manager/0.log"
Sep 16 14:22:49.065648 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:49.065620    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5fcfb689-h8t7z_bb874d95-6b2b-41eb-acab-aaff3c882a3e/copy-manifests/0.log"
Sep 16 14:22:50.422948 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:50.422919    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-66dmq_ae21a4ff-f37f-43cd-876d-1351113bb384/istio-proxy/0.log"
Sep 16 14:22:50.540594 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:50.540518    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6465fbddb8-dxpw8_5b2df2b7-b873-42cf-a6a0-84be2e90c057/manager/0.log"
Sep 16 14:22:51.276649 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:51.276616    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-l5hkn_1ce50390-e6e1-49fe-9e01-8e74baea14ac/istio-proxy/0.log"
Sep 16 14:22:51.283784 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:51.283761    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-l5hkn_1ce50390-e6e1-49fe-9e01-8e74baea14ac/istio-proxy/1.log"
Sep 16 14:22:51.321547 ip-10-0-129-226 kubenswrapper[2548]: I0916 14:22:51.321519    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wzccf_01a23ec2-ae3a-4d16-95b2-8ed7ede19084/payload-pre-processing/0.log"