Sep 08 13:26:16.576598 ip-10-0-130-208 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 13:26:16.576612 ip-10-0-130-208 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 13:26:16.576623 ip-10-0-130-208 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 13:26:16.576943 ip-10-0-130-208 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 13:26:27.153793 ip-10-0-130-208 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 13:26:27.153893 ip-10-0-130-208 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5d133f93f6ab44ba9ab78e1301731059 --
Sep 08 13:28:11.523645 ip-10-0-130-208 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:28:11.965336 ip-10-0-130-208 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:28:11.965336 ip-10-0-130-208 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:28:11.965336 ip-10-0-130-208 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:28:11.965336 ip-10-0-130-208 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:28:11.965336 ip-10-0-130-208 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:28:11.966444 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.966308    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:28:11.972411 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972392    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:28:11.972411 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972407    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:28:11.972411 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972412    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:28:11.972411 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972416    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972420    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972425    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972435    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972439    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972443    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972446    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972450    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972454    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972458    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972499    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972505    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972510    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972514    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972521    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972527    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972532    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972536    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972540    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:28:11.972646 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972544    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972549    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972553    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972557    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972562    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972566    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972573    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972578    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972584    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972589    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972593    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972598    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972602    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972606    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972610    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972616    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972621    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972626    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972631    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:28:11.973456 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972636    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972640    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972645    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972649    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972653    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972657    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972661    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972665    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972669    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972674    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972678    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972682    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972687    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972691    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972697    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972702    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972706    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972710    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972715    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972719    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:28:11.974250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972724    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972728    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972733    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972738    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972742    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972746    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972750    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972755    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972760    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972764    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972768    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972773    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972778    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972783    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972787    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972791    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972796    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972800    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972806    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972810    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:28:11.975094 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972816    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972820    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972824    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972828    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972832    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972837    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972840    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972845    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972849    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972854    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972858    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972864    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972870    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972875    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972879    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972883    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972887    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.972891    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974038    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:28:11.976022 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974049    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974056    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974061    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974066    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974071    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974076    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974081    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974085    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974090    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974094    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974099    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974103    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974107    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974111    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974116    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974120    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974124    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974128    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974132    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:28:11.976843 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974137    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974141    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974145    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974150    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974154    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974158    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974162    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974166    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974171    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974174    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974179    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974183    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974187    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974192    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974196    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974201    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974205    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974209    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974213    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974217    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:28:11.977562 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974222    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974226    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974230    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974234    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974239    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974242    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974247    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974251    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974255    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974259    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974262    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974266    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974270    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974274    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974278    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974282    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974286    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974291    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974295    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974299    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:28:11.978069 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974303    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974307    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974313    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974317    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974321    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974326    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974331    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974335    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974340    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974344    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974348    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974352    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974357    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974361    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974366    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974370    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974374    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974397    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974403    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974408    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:28:11.978583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974414    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974420    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974424    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974428    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974433    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974437    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974442    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974448    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974453    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974459    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974465    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974471    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974476    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974481    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974486    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974490    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974494    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974499    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.974504    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:28:11.979090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975125    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975135    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975141    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975145    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975149    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975155    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975160    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975165    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975173    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975179    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975184    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975189    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975194    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975198    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975203    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975207    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975211    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975215    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975219    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:28:11.979647 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975224    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975228    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975233    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975239    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975245    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975249    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975254    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975258    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975263    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975266    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975271    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975275    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975279    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975283    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975288    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975292    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975296    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975300    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975304    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975309    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:28:11.980141 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975313    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975317    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975322    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975326    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975330    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975334    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975338    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975342    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975346    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975350    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975354    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975361    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975367    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975371    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975394    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975398    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975402    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975406    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975411    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975415    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:28:11.980652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975418    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975423    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975427    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975432    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975437    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975441    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975445    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975449    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975453    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975456    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975461    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975465    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975470    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975474    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975479    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975483    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975487    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975491    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975495    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975502    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:28:11.981420 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975506    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975510    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975514    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975518    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975526    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975532    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975537    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975542    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975546    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975550    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975557    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975561    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975565    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975569    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975574    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975578    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975582    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975586    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975590    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.975595    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:28:11.981906 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975683    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975730    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975740    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975747    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975754    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975760    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975766    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975772    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975777    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975782    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975788    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975795    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975800    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975805    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975810    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975814    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975821    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975826    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975832    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975837    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975842    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975847    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975852    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975859    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:28:11.982439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975864    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975869    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975875    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975880    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975885    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975890    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975895    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975900    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975906    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975911    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975916    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975921    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975925    2561 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975930    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975938    2561 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975943    2561 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975948    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975953    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975958    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975965    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975970    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975975    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975980    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975985    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975991    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:28:11.983068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.975996    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976001    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976005    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976010    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976014    2561 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976020    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976024    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976029    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976035    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976040    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976045    2561 flags.go:64] FLAG: --help="false"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976050    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-130-208.ec2.internal"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976055    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976060    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976065    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976071    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976077    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976082    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976086    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976091    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976096    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976101    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976107    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976112    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:28:11.983715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976117    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976122    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976128    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976133    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976137    2561 flags.go:64] FLAG: --lock-file=""
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976142    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976147    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976154    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976162    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976166    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976171    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976175    2561 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976180    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976186    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976191    2561 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976196    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976203    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976209    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976215    2561 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976219    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976224    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976229    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976234    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976238    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976243    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:28:11.984297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976248    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976259    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976264    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976270    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976274    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976279    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976288    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976293    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976297    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976304    2561 flags.go:64] FLAG: --port="10250"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976309    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976314    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-086ba0c2153ad579e"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976320    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976324    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976329    2561 flags.go:64] FLAG: --register-node="true"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976335    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976341    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976346    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976351    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976355    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976361    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976366    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976371    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976391    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976397    2561 flags.go:64] FLAG: --runonce="false"
Sep 08 13:28:11.984952 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976401    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976407    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976412    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976417    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976421    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976426    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976431    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976437    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976441    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976446    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976451    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976456    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976461    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976466    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976472    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976480    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976487    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976492    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976499    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976504    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976509    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976514    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976520    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976525    2561 flags.go:64] FLAG: --v="2"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976531    2561 flags.go:64] FLAG: --version="false"
Sep 08 13:28:11.985754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976537    2561 flags.go:64] FLAG: --vmodule=""
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976543    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.976548    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976738    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976744    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976749    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976754    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976758    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976763    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976769    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976774    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976779    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976784    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976788    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976793    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976797    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976802    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976806    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976811    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976816    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:28:11.986727 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976822    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976828    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976833    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976840    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976844    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976848    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976852    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976856    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976860    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976866    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976870    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976875    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976879    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976884    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976888    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976892    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976897    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976901    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976905    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:28:11.987366 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976909    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976914    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976918    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976922    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976926    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976931    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976936    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976940    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976944    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976948    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976952    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976956    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976961    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976965    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976969    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976973    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976983    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976988    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976992    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.976996    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:28:11.987847 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977000    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977004    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977011    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977015    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977019    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977023    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977028    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977032    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977035    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977039    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977043    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977049    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977055    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977059    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977063    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977067    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977071    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977076    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977082    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:28:11.988320 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977087    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977092    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977096    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977101    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977105    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977110    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977114    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977118    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977123    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977129    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977134    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977138    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977142    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977146    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977150    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977156    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977160    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977164    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977168    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977173    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:28:11.988802 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977178    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977190    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977194    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.977198    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.977206    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.988011    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.988025    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.988045    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.988050    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988119    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988124    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988128    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988131    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988133    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988136    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:28:11.989337 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988140    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988144    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988147    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988150    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988152    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988156    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988159    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988162    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988164    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988167    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988169    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988172    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988175    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988186    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988189    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988192    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988194    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988197    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988200    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988202    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:28:11.989741 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988205    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988208    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988211    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988214    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988217    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988219    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988222    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988225    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988227    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988230    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988232    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988235    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988238    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988240    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988242    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988245    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988247    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988250    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988252    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:28:11.990215 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988256    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988261    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988265    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988269    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988271    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988274    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988277    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988280    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988283    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988286    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988289    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988292    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988294    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988297    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988300    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988302    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988305    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988308    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988310    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988313    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:28:11.990697 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988316    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988318    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988321    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988323    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988326    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988328    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988331    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988334    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988336    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988339    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988341    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988344    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988346    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988349    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988352    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988355    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988358    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988360    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988363    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988365    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:28:11.991185 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988368    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988370    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988373    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988391    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988395    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988398    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988401    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988403    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988406    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988409    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988411    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988414    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988417    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988419    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:28:11.991751 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.988424    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988536    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988540    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988544    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988547    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988550    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988553    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988556    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988559    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988562    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988565    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988567    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988570    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988574    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988576    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988579    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988582    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988586    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988590    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:28:11.992090 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988593    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988597    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988600    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988603    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988606    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988609    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988612    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988614    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988617    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988619    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988622    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988625    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988627    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988630    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988634    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988638    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988642    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988646    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988650    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:28:11.992605 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988654    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988657    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988660    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988663    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988665    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988668    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988670    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988673    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988677    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988680    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988682    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988685    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988688    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988690    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988693    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988696    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988699    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988701    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988704    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988707    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:28:11.993063 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988709    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988712    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988715    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988717    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988720    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988722    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988725    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988728    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988732    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988734    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988737    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988739    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988742    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988744    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988747    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988749    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988752    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988754    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988757    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:28:11.993576 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988759    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988762    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988765    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988767    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988770    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988772    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988775    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988777    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988780    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988783    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988785    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988788    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988791    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988793    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988795    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988798    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988800    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988803    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988806    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988809    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:28:11.994021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988811    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:28:11.994507 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988814    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:28:11.994507 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:11.988816    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:28:11.994507 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.988821    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:28:11.994507 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.988922    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:28:11.994507 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.992172    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:28:11.994507 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.993294    2561 server.go:1019] "Starting client certificate rotation"
Sep 08 13:28:11.994507 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.993403    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:28:11.994507 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.993433    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:28:11.996198 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.996185    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:28:11.997265 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:11.997248    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:28:12.006089 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.006072    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:28:12.009973 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.009959    2561 log.go:25] "Validated CRI v1 image API"
Sep 08 13:28:12.013195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.013175    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:28:12.016675 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.016658    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 850ae284-a15a-49dc-99d8-ce065608f2ab:/dev/nvme0n1p4 dcef49ef-0010-4104-bd9c-1ca46724e947:/dev/nvme0n1p3]
Sep 08 13:28:12.016735 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.016680    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:28:12.023441 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.023422    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:28:12.023585 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.023469    2561 manager.go:216] Machine: {Timestamp:2026-09-08 13:28:12.021960834 +0000 UTC m=+0.372946793 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098902 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2668dc36dd6ce795770ff46107b839 SystemUUID:ec2668dc-36dd-6ce7-9577-0ff46107b839 BootID:5d133f93-f6ab-44ba-9ab7-8e1301731059 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:71:ed:3b:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:71:ed:3b:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:58:0c:03:38:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:28:12.023585 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.023582    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:28:12.023673 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.023661    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:28:12.027172 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.027145    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:28:12.027302 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.027175    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-208.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:28:12.028006 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.027996    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:28:12.028038 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.028009    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:28:12.028038 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.028016    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:28:12.028825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.028815    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:28:12.030723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.030713    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:28:12.034196 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034185    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:28:12.034299 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034290    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:28:12.034335 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034319    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:28:12.034335 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034328    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:28:12.034403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034341    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:28:12.034403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034350    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:28:12.034403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034359    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:28:12.034759 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034747    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:28:12.034799 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.034766    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:28:12.037025 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.037009    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:28:12.040001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.039989    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:28:12.040044 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040018    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:28:12.040196 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040182    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:28:12.040226 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040209    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:28:12.040226 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040219    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:28:12.040274 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040226    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:28:12.040274 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040238    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:28:12.040274 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040247    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:28:12.040812 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040801    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:28:12.040851 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040813    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:28:12.040851 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040821    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:28:12.040851 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040827    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:28:12.040851 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040839    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:28:12.040851 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.040847    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:28:12.041737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.041725    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:28:12.041769 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.041739    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:28:12.045207 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.045063    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-208.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:28:12.045307 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.045250    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-208.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:28:12.045307 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.045250    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:28:12.045738 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.045723    2561 server.go:1268] "Started kubelet"
Sep 08 13:28:12.045825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.045796    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:28:12.046001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.045946    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:28:12.046097 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.046022    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:28:12.046779 ip-10-0-130-208 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:28:12.047743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.047368    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:28:12.047743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.047446    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:28:12.050458 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.050440    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:28:12.054162 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.054143    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:28:12.054855 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.054838    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:28:12.055011 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.054989    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.055579 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.055563    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:28:12.055579 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.055582    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:28:12.055707 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.055562    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:28:12.059082 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.059050    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:28:12.059082 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.059080    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:28:12.059214 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.054812    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-208.ec2.internal.18d35b8244487b3a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-208.ec2.internal,UID:ip-10-0-130-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-208.ec2.internal,},FirstTimestamp:2026-09-08 13:28:12.045695802 +0000 UTC m=+0.396681741,LastTimestamp:2026-09-08 13:28:12.045695802 +0000 UTC m=+0.396681741,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-208.ec2.internal,}"
Sep 08 13:28:12.059493 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.059475    2561 factory.go:55] Registering systemd factory
Sep 08 13:28:12.059649 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.059635    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:28:12.059928 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.059912    2561 factory.go:153] Registering CRI-O factory
Sep 08 13:28:12.060011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.059932    2561 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:28:12.060011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.060001    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:28:12.060112 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.060022    2561 factory.go:103] Registering Raw factory
Sep 08 13:28:12.060112 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.060049    2561 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:28:12.061269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.061248    2561 manager.go:349] Starting recovery of all containers
Sep 08 13:28:12.069738 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.069052    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qzdx9"
Sep 08 13:28:12.069738 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.069093    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:28:12.069738 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.069152    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-208.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:28:12.072449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.072431    2561 manager.go:354] Recovery completed
Sep 08 13:28:12.076543 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.076525    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qzdx9"
Sep 08 13:28:12.076735 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.076723    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:28:12.078577 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.078561    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:28:12.078644 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.078588    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:28:12.078644 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.078599    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:28:12.080853 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.080217    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:28:12.080853 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.080232    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:28:12.080853 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.080249    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:28:12.080853 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.080708    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-208.ec2.internal.18d35b82463e3177  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-208.ec2.internal,UID:ip-10-0-130-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-208.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-208.ec2.internal,},FirstTimestamp:2026-09-08 13:28:12.078575991 +0000 UTC m=+0.429561932,LastTimestamp:2026-09-08 13:28:12.078575991 +0000 UTC m=+0.429561932,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-208.ec2.internal,}"
Sep 08 13:28:12.081068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.080930    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:28:12.081621 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.081609    2561 policy_none.go:49] "None policy: Start"
Sep 08 13:28:12.081665 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.081624    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:28:12.081665 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.081634    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:28:12.082544 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.082531    2561 policy_none.go:47] "Start"
Sep 08 13:28:12.083063 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.083048    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:28:12.083152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.083075    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:28:12.083152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.083089    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:28:12.117540 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.117524    2561 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:28:12.117619 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.117562    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:28:12.117619 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.117572    2561 server.go:87] "Starting device plugin registration server"
Sep 08 13:28:12.117889 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.117876    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:28:12.117946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.117896    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:28:12.117989 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.117977    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:28:12.118062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.118049    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:28:12.118155 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.118063    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:28:12.118664 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.118645    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:28:12.118731 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.118698    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.160449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.159559    2561 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2594 into the system container \"/system.slice\": <nil>"
Sep 08 13:28:12.218114 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.218040    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:28:12.218964 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.218947    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:28:12.219040 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.218981    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:28:12.219040 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.218994    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:28:12.219040 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.219024    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.228080 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.228064    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.231694 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.228086    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-208.ec2.internal\": node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.237593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.237573    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:28:12.238731 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.238713    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:28:12.238815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.238754    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:28:12.238815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.238773    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:28:12.238815 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.238803    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:28:12.240914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.240884    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:28:12.249926 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.249909    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.339923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.339880    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal"]
Sep 08 13:28:12.340078 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.339960    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:28:12.342601 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.342586    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:28:12.342684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.342617    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:28:12.342684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.342634    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:28:12.343731 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.343716    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:28:12.343844 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.343820    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.343890 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.343849    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:28:12.344685 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.344666    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:28:12.344757 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.344698    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:28:12.344757 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.344712    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:28:12.344757 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.344671    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:28:12.344880 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.344763    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:28:12.344880 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.344773    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:28:12.345740 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.345725    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.345810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.345751    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:28:12.348247 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.348230    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:28:12.348322 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.348255    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:28:12.348322 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.348265    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:28:12.350026 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.349965    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.360609 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.360588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9135ca1286983be48fef1a9d4c6b86dc-config\") pod \"kube-apiserver-proxy-ip-10-0-130-208.ec2.internal\" (UID: \"9135ca1286983be48fef1a9d4c6b86dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.360684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.360616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a1d821425d1dafec7de35f4d65d5b252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal\" (UID: \"a1d821425d1dafec7de35f4d65d5b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.360684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.360642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d821425d1dafec7de35f4d65d5b252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal\" (UID: \"a1d821425d1dafec7de35f4d65d5b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.371942 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.371921    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-208.ec2.internal\" not found" node="ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.376198 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.376184    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-208.ec2.internal\" not found" node="ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.451073 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.451050    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.461471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.461447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a1d821425d1dafec7de35f4d65d5b252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal\" (UID: \"a1d821425d1dafec7de35f4d65d5b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.461555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.461481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d821425d1dafec7de35f4d65d5b252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal\" (UID: \"a1d821425d1dafec7de35f4d65d5b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.461555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.461506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9135ca1286983be48fef1a9d4c6b86dc-config\") pod \"kube-apiserver-proxy-ip-10-0-130-208.ec2.internal\" (UID: \"9135ca1286983be48fef1a9d4c6b86dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.461555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.461539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9135ca1286983be48fef1a9d4c6b86dc-config\") pod \"kube-apiserver-proxy-ip-10-0-130-208.ec2.internal\" (UID: \"9135ca1286983be48fef1a9d4c6b86dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.461555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.461539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a1d821425d1dafec7de35f4d65d5b252-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal\" (UID: \"a1d821425d1dafec7de35f4d65d5b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.461555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.461546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d821425d1dafec7de35f4d65d5b252-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal\" (UID: \"a1d821425d1dafec7de35f4d65d5b252\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.551905 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.551836    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.652352 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.652317    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.675849 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.675825    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.680127 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.679998    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:12.753189 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.753143    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.855777 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.855737    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.894857 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.894830    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:28:12.956847 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:12.956819    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:12.993956 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.993928    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:28:12.994351 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.994078    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:28:12.994351 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:12.994106    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:28:13.057297 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:13.057268    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:13.072484 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.072413    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:28:13.078865 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.078837    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:23:12 +0000 UTC" deadline="2028-04-04 13:38:22.99032708 +0000 UTC"
Sep 08 13:28:13.078933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.078865    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13776h10m9.911465167s"
Sep 08 13:28:13.158108 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:13.158085    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:13.259079 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:13.259055    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:13.361939 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:13.361860    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:13.462988 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:13.462961    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:13.563256 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:13.563218    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-208.ec2.internal\" not found"
Sep 08 13:28:13.602545 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.602517    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:28:13.659065 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.658996    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:13.672620 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.672592    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:28:13.673861 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.673848    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal"
Sep 08 13:28:13.682375 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.682357    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:28:13.903747 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:13.903730    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:28:14.037634 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.037537    2561 apiserver.go:52] "Watching apiserver"
Sep 08 13:28:14.044973 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.044950    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:28:14.046120 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.046098    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-pl7gt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bct9v","openshift-image-registry/node-ca-hdfv6","openshift-multus/multus-additional-cni-plugins-n8vsw","openshift-multus/multus-rjgp2","openshift-multus/network-metrics-daemon-zspg2","openshift-network-diagnostics/network-check-target-gf4f5","openshift-network-operator/iptables-alerter-vr5qx","openshift-ovn-kubernetes/ovnkube-node-5bjbc","kube-system/global-pull-secret-syncer-wcdnb","kube-system/konnectivity-agent-q7z85","kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"]
Sep 08 13:28:14.047928 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.047913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:14.048007 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.047980    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:14.049432 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.049168    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:14.051524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.051503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.051675 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.051656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.051977 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.051953    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:28:14.052107 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.051988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gqcmx\""
Sep 08 13:28:14.052268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.052249    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:28:14.052691 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.052673    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.053725 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.053696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.053992 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.053973    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:28:14.054095 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.054026    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:28:14.054177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.054161    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:28:14.054357 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.054341    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pqgsm\""
Sep 08 13:28:14.054357 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.054355    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5mhlk\""
Sep 08 13:28:14.054538 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.054523    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:28:14.054596 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.054578    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:28:14.054795 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.054780    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:28:14.055292 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.055271    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:28:14.055463 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.055444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.055540 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.055453    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9qg79\""
Sep 08 13:28:14.055598 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.055576    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:28:14.055743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.055718    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:28:14.055851 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.055819    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:28:14.056042 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.056023    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:28:14.056233 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.056198    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dkk6g\""
Sep 08 13:28:14.056317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.056212    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:28:14.056688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.056672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.057726 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.057685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.057823 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.057794    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:14.058291 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.058270    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:28:14.058394 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.058328    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:28:14.058394 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.058334    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bfk8r\""
Sep 08 13:28:14.058394 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.058359    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:28:14.059296 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.058967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.059296 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.059120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:28:14.059444 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.059324    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzt9g\""
Sep 08 13:28:14.059444 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.059413    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:28:14.059582 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.059533    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:28:14.060141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.060126    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.061309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.061289    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:14.061392 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.061361    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:14.061476 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.061389    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:28:14.061633 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.061614    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:28:14.061793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.061773    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8mjtt\""
Sep 08 13:28:14.061793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.061786    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:28:14.062118 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.062102    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:28:14.062200 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.062100    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:28:14.062258 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.062197    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:28:14.062258 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.062204    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:28:14.062638 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.062621    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-22pgs\""
Sep 08 13:28:14.062736 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.062717    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:28:14.072436 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5pbx\" (UniqueName: \"kubernetes.io/projected/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-kube-api-access-c5pbx\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.072547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85e17bd3-c3e9-4b85-b945-e55797a49b90-iptables-alerter-script\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.072547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysctl-d\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.072547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjxr\" (UniqueName: \"kubernetes.io/projected/902e01bc-c36e-4c70-be62-390a29c4697f-kube-api-access-znjxr\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.072547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.072741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-netns\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.072741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-cni-bin\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.072741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:14.072741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.072741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-multus-certs\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.072741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.072950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-modprobe-d\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.072950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-cnibin\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.072950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgmb\" (UniqueName: \"kubernetes.io/projected/f3cc2402-36f3-4342-8d14-e978f9e66436-kube-api-access-9tgmb\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.072950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-daemon-config\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.072950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-device-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.072950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38ef716f-8b3c-4b09-b816-1e59fdefc265-agent-certs\") pod \"konnectivity-agent-q7z85\" (UID: \"38ef716f-8b3c-4b09-b816-1e59fdefc265\") " pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:14.072950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysctl-conf\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.072950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91b8aa20-62a5-4327-ba6f-563710623ce6-serviceca\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.072992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-kubelet\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-sys-fs\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/902e01bc-c36e-4c70-be62-390a29c4697f-tmp\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-hostroot\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-var-lib-kubelet\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-k8s-cni-cncf-io\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10415704-0e6c-43d8-b16f-c5a2ee365b5e-cni-binary-copy\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.073269 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-socket-dir-parent\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38ef716f-8b3c-4b09-b816-1e59fdefc265-konnectivity-ca\") pod \"konnectivity-agent-q7z85\" (UID: \"38ef716f-8b3c-4b09-b816-1e59fdefc265\") " pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073321    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5vk\" (UniqueName: \"kubernetes.io/projected/6404c150-528e-4903-8049-9fa576f8f9ea-kube-api-access-bz5vk\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysconfig\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/902e01bc-c36e-4c70-be62-390a29c4697f-etc-tuned\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-socket-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-system-cni-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6jz\" (UniqueName: \"kubernetes.io/projected/85e17bd3-c3e9-4b85-b945-e55797a49b90-kube-api-access-5z6jz\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-registration-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-sys\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073610    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-lib-modules\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-host\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.073705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-kubernetes\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx28r\" (UniqueName: \"kubernetes.io/projected/91b8aa20-62a5-4327-ba6f-563710623ce6-kube-api-access-vx28r\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-cni-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-cnibin\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-conf-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91b8aa20-62a5-4327-ba6f-563710623ce6-host\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-system-cni-dir\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-etc-kubernetes\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5ppw\" (UniqueName: \"kubernetes.io/projected/10415704-0e6c-43d8-b16f-c5a2ee365b5e-kube-api-access-z5ppw\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-systemd\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-run\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.073962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-cni-multus\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.074130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.074005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85e17bd3-c3e9-4b85-b945-e55797a49b90-host-slash\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.092425 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.092407    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttxg6"
Sep 08 13:28:14.101237 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.101220    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttxg6"
Sep 08 13:28:14.157064 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.157036    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:28:14.174219 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.174406 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.174406 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-kubelet\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.174406 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.174406 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.174406 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-sys-fs\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.174406 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-kubelet\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-sys-fs\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f67904d-8e19-44ee-a8ed-9da81a313102-tmp-dir\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-run-netns\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/902e01bc-c36e-4c70-be62-390a29c4697f-tmp\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-hostroot\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04639862-2bd6-4c98-896e-473ccb2d7bd7-kubelet-config\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-log-socket\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-var-lib-kubelet\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174724    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-hostroot\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-k8s-cni-cncf-io\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.174771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04639862-2bd6-4c98-896e-473ccb2d7bd7-dbus\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qspvf\" (UniqueName: \"kubernetes.io/projected/9f67904d-8e19-44ee-a8ed-9da81a313102-kube-api-access-qspvf\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-var-lib-kubelet\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-var-lib-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-k8s-cni-cncf-io\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-etc-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10415704-0e6c-43d8-b16f-c5a2ee365b5e-cni-binary-copy\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-socket-dir-parent\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38ef716f-8b3c-4b09-b816-1e59fdefc265-konnectivity-ca\") pod \"konnectivity-agent-q7z85\" (UID: \"38ef716f-8b3c-4b09-b816-1e59fdefc265\") " pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-socket-dir-parent\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.174994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f67904d-8e19-44ee-a8ed-9da81a313102-hosts-file\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-node-log\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5vk\" (UniqueName: \"kubernetes.io/projected/6404c150-528e-4903-8049-9fa576f8f9ea-kube-api-access-bz5vk\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysconfig\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/902e01bc-c36e-4c70-be62-390a29c4697f-etc-tuned\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.175340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175134    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-socket-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-system-cni-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6jz\" (UniqueName: \"kubernetes.io/projected/85e17bd3-c3e9-4b85-b945-e55797a49b90-kube-api-access-5z6jz\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-registration-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175259    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-slash\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-cni-bin\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrbm\" (UniqueName: \"kubernetes.io/projected/890f71c6-95de-42c3-9790-4d3c7155687d-kube-api-access-nwrbm\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-sys\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-lib-modules\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-host\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-registration-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysconfig\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-sys\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175136    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-host\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-system-cni-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-socket-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:14.176062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-systemd\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-lib-modules\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-kubernetes\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx28r\" (UniqueName: \"kubernetes.io/projected/91b8aa20-62a5-4327-ba6f-563710623ce6-kube-api-access-vx28r\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-kubernetes\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-cni-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-cnibin\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-conf-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-systemd-units\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91b8aa20-62a5-4327-ba6f-563710623ce6-host\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-system-cni-dir\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-cnibin\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-etc-kubernetes\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5ppw\" (UniqueName: \"kubernetes.io/projected/10415704-0e6c-43d8-b16f-c5a2ee365b5e-kube-api-access-z5ppw\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175945    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-etc-kubernetes\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-systemd\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-cni-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91b8aa20-62a5-4327-ba6f-563710623ce6-host\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.176813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.175996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-run\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-cni-multus\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-system-cni-dir\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-systemd\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85e17bd3-c3e9-4b85-b945-e55797a49b90-host-slash\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-kubelet\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-conf-dir\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-ovn\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-cni-multus\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-run\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85e17bd3-c3e9-4b85-b945-e55797a49b90-host-slash\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5pbx\" (UniqueName: \"kubernetes.io/projected/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-kube-api-access-c5pbx\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-cni-netd\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10415704-0e6c-43d8-b16f-c5a2ee365b5e-cni-binary-copy\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/890f71c6-95de-42c3-9790-4d3c7155687d-ovn-node-metrics-cert\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85e17bd3-c3e9-4b85-b945-e55797a49b90-iptables-alerter-script\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.177663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysctl-d\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjxr\" (UniqueName: \"kubernetes.io/projected/902e01bc-c36e-4c70-be62-390a29c4697f-kube-api-access-znjxr\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysctl-d\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-netns\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/38ef716f-8b3c-4b09-b816-1e59fdefc265-konnectivity-ca\") pod \"konnectivity-agent-q7z85\" (UID: \"38ef716f-8b3c-4b09-b816-1e59fdefc265\") " pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-netns\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-cni-bin\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-ovnkube-script-lib\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-multus-certs\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-var-lib-cni-bin\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.176792    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-modprobe-d\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-cnibin\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.178555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85e17bd3-c3e9-4b85-b945-e55797a49b90-iptables-alerter-script\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.176892    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:28:14.676851506 +0000 UTC m=+3.027837436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgmb\" (UniqueName: \"kubernetes.io/projected/f3cc2402-36f3-4342-8d14-e978f9e66436-kube-api-access-9tgmb\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-daemon-config\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/10415704-0e6c-43d8-b16f-c5a2ee365b5e-host-run-multus-certs\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-device-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.176998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-ovnkube-config\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3cc2402-36f3-4342-8d14-e978f9e66436-cnibin\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-env-overrides\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38ef716f-8b3c-4b09-b816-1e59fdefc265-agent-certs\") pod \"konnectivity-agent-q7z85\" (UID: \"38ef716f-8b3c-4b09-b816-1e59fdefc265\") " pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-device-dir\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysctl-conf\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91b8aa20-62a5-4327-ba6f-563710623ce6-serviceca\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3cc2402-36f3-4342-8d14-e978f9e66436-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.179528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177332    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-modprobe-d\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.180028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902e01bc-c36e-4c70-be62-390a29c4697f-etc-sysctl-conf\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.180028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177501    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/10415704-0e6c-43d8-b16f-c5a2ee365b5e-multus-daemon-config\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.180028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.177691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91b8aa20-62a5-4327-ba6f-563710623ce6-serviceca\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.180028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.179595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/902e01bc-c36e-4c70-be62-390a29c4697f-etc-tuned\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.180028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.179676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/902e01bc-c36e-4c70-be62-390a29c4697f-tmp\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.180028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.179781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/38ef716f-8b3c-4b09-b816-1e59fdefc265-agent-certs\") pod \"konnectivity-agent-q7z85\" (UID: \"38ef716f-8b3c-4b09-b816-1e59fdefc265\") " pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:14.187869 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.187731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5pbx\" (UniqueName: \"kubernetes.io/projected/5462eacf-bac5-4bb8-9e6d-612d53bc8aa4-kube-api-access-c5pbx\") pod \"aws-ebs-csi-driver-node-gp9wt\" (UID: \"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.187869 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.187798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5ppw\" (UniqueName: \"kubernetes.io/projected/10415704-0e6c-43d8-b16f-c5a2ee365b5e-kube-api-access-z5ppw\") pod \"multus-rjgp2\" (UID: \"10415704-0e6c-43d8-b16f-c5a2ee365b5e\") " pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.188084 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.188059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5vk\" (UniqueName: \"kubernetes.io/projected/6404c150-528e-4903-8049-9fa576f8f9ea-kube-api-access-bz5vk\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:14.188149 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.188116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjxr\" (UniqueName: \"kubernetes.io/projected/902e01bc-c36e-4c70-be62-390a29c4697f-kube-api-access-znjxr\") pod \"tuned-bct9v\" (UID: \"902e01bc-c36e-4c70-be62-390a29c4697f\") " pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.188438 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.188418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx28r\" (UniqueName: \"kubernetes.io/projected/91b8aa20-62a5-4327-ba6f-563710623ce6-kube-api-access-vx28r\") pod \"node-ca-hdfv6\" (UID: \"91b8aa20-62a5-4327-ba6f-563710623ce6\") " pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.188632 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.188614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgmb\" (UniqueName: \"kubernetes.io/projected/f3cc2402-36f3-4342-8d14-e978f9e66436-kube-api-access-9tgmb\") pod \"multus-additional-cni-plugins-n8vsw\" (UID: \"f3cc2402-36f3-4342-8d14-e978f9e66436\") " pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.189227 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.189209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6jz\" (UniqueName: \"kubernetes.io/projected/85e17bd3-c3e9-4b85-b945-e55797a49b90-kube-api-access-5z6jz\") pod \"iptables-alerter-vr5qx\" (UID: \"85e17bd3-c3e9-4b85-b945-e55797a49b90\") " pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.246829 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.246777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal" event={"ID":"a1d821425d1dafec7de35f4d65d5b252","Type":"ContainerStarted","Data":"6f4acc5432bd9b5105857c7af3a09a0886abad7a8c3089fd50ee184115cc842d"}
Sep 08 13:28:14.278331 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-cni-bin\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278331 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrbm\" (UniqueName: \"kubernetes.io/projected/890f71c6-95de-42c3-9790-4d3c7155687d-kube-api-access-nwrbm\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-systemd\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-systemd-units\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-cni-bin\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-kubelet\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-ovn\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-systemd\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-cni-netd\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.278524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-systemd-units\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278542    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-ovn\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/890f71c6-95de-42c3-9790-4d3c7155687d-ovn-node-metrics-cert\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-cni-netd\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.278648    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-kubelet\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.278696    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret podName:04639862-2bd6-4c98-896e-473ccb2d7bd7 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:14.778678263 +0000 UTC m=+3.129664196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret") pod "global-pull-secret-syncer-wcdnb" (UID: "04639862-2bd6-4c98-896e-473ccb2d7bd7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-ovnkube-script-lib\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-ovnkube-config\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-env-overrides\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f67904d-8e19-44ee-a8ed-9da81a313102-tmp-dir\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-run-netns\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04639862-2bd6-4c98-896e-473ccb2d7bd7-kubelet-config\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.279011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-run-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-log-socket\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278969    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04639862-2bd6-4c98-896e-473ccb2d7bd7-dbus\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-run-netns\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.278997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qspvf\" (UniqueName: \"kubernetes.io/projected/9f67904d-8e19-44ee-a8ed-9da81a313102-kube-api-access-qspvf\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-var-lib-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-log-socket\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-etc-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279074    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04639862-2bd6-4c98-896e-473ccb2d7bd7-kubelet-config\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f67904d-8e19-44ee-a8ed-9da81a313102-hosts-file\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-node-log\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04639862-2bd6-4c98-896e-473ccb2d7bd7-dbus\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-slash\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f67904d-8e19-44ee-a8ed-9da81a313102-hosts-file\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279124    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-env-overrides\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279320    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-ovnkube-script-lib\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.279718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-node-log\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.280248 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f67904d-8e19-44ee-a8ed-9da81a313102-tmp-dir\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.280248 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-etc-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.280248 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/890f71c6-95de-42c3-9790-4d3c7155687d-ovnkube-config\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.280248 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-host-slash\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.280248 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.279354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/890f71c6-95de-42c3-9790-4d3c7155687d-var-lib-openvswitch\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.280925 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.280906    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/890f71c6-95de-42c3-9790-4d3c7155687d-ovn-node-metrics-cert\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.284622 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.284603    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:28:14.284676 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.284628    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:28:14.284676 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.284643    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fgv2b for pod openshift-network-diagnostics/network-check-target-gf4f5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:14.284746 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.284713    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b podName:ad8bf542-ec0c-429c-8d95-40774e89f90c nodeName:}" failed. No retries permitted until 2026-09-08 13:28:14.78469691 +0000 UTC m=+3.135682876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fgv2b" (UniqueName: "kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b") pod "network-check-target-gf4f5" (UID: "ad8bf542-ec0c-429c-8d95-40774e89f90c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:14.286818 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.286799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrbm\" (UniqueName: \"kubernetes.io/projected/890f71c6-95de-42c3-9790-4d3c7155687d-kube-api-access-nwrbm\") pod \"ovnkube-node-5bjbc\" (UID: \"890f71c6-95de-42c3-9790-4d3c7155687d\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.287128 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.287112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qspvf\" (UniqueName: \"kubernetes.io/projected/9f67904d-8e19-44ee-a8ed-9da81a313102-kube-api-access-qspvf\") pod \"node-resolver-pl7gt\" (UID: \"9f67904d-8e19-44ee-a8ed-9da81a313102\") " pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.362492 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.362370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:14.368191 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.368173    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bct9v"
Sep 08 13:28:14.371425 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.371395    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ef716f_8b3c_4b09_b816_1e59fdefc265.slice/crio-0cd7b4698694282d05c70f33de6092aa57aeb2ef60b3f942dc252dbd6d44db9f WatchSource:0}: Error finding container 0cd7b4698694282d05c70f33de6092aa57aeb2ef60b3f942dc252dbd6d44db9f: Status 404 returned error can't find the container with id 0cd7b4698694282d05c70f33de6092aa57aeb2ef60b3f942dc252dbd6d44db9f
Sep 08 13:28:14.375840 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.375817    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902e01bc_c36e_4c70_be62_390a29c4697f.slice/crio-e0cc1969996023707ea5dce6bfa6d36b5dbcef9478c65053e0226c6bf2129be8 WatchSource:0}: Error finding container e0cc1969996023707ea5dce6bfa6d36b5dbcef9478c65053e0226c6bf2129be8: Status 404 returned error can't find the container with id e0cc1969996023707ea5dce6bfa6d36b5dbcef9478c65053e0226c6bf2129be8
Sep 08 13:28:14.376651 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.376635    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hdfv6"
Sep 08 13:28:14.382331 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.382313    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n8vsw"
Sep 08 13:28:14.384172 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.384149    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b8aa20_62a5_4327_ba6f_563710623ce6.slice/crio-5d28bbdb2e05d8c769988fa6ccb225e969bf7ce415145d4d5ecc30c930360121 WatchSource:0}: Error finding container 5d28bbdb2e05d8c769988fa6ccb225e969bf7ce415145d4d5ecc30c930360121: Status 404 returned error can't find the container with id 5d28bbdb2e05d8c769988fa6ccb225e969bf7ce415145d4d5ecc30c930360121
Sep 08 13:28:14.389234 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.389210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rjgp2"
Sep 08 13:28:14.389435 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.389416    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cc2402_36f3_4342_8d14_e978f9e66436.slice/crio-ea8696e16e1e89aa5af0a7fb9c78a33768602801b6788d6ec0468dfdd0c9381f WatchSource:0}: Error finding container ea8696e16e1e89aa5af0a7fb9c78a33768602801b6788d6ec0468dfdd0c9381f: Status 404 returned error can't find the container with id ea8696e16e1e89aa5af0a7fb9c78a33768602801b6788d6ec0468dfdd0c9381f
Sep 08 13:28:14.396189 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.396171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vr5qx"
Sep 08 13:28:14.398243 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.398220    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10415704_0e6c_43d8_b16f_c5a2ee365b5e.slice/crio-6cb3ca568d7c5b66623813f96569655d78190f87bb2339b2b4368e8477fe2132 WatchSource:0}: Error finding container 6cb3ca568d7c5b66623813f96569655d78190f87bb2339b2b4368e8477fe2132: Status 404 returned error can't find the container with id 6cb3ca568d7c5b66623813f96569655d78190f87bb2339b2b4368e8477fe2132
Sep 08 13:28:14.401391 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.401369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt"
Sep 08 13:28:14.403082 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.403062    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e17bd3_c3e9_4b85_b945_e55797a49b90.slice/crio-80cb5e448e307b15a45c98a882b648ac015e1548239e53364edfc2c83d0d860d WatchSource:0}: Error finding container 80cb5e448e307b15a45c98a882b648ac015e1548239e53364edfc2c83d0d860d: Status 404 returned error can't find the container with id 80cb5e448e307b15a45c98a882b648ac015e1548239e53364edfc2c83d0d860d
Sep 08 13:28:14.407618 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.407597    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:14.409764 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.409745    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5462eacf_bac5_4bb8_9e6d_612d53bc8aa4.slice/crio-a97420c0c64a60bc1911e6552da271cd6f19a87abaebd4e0f101b30ca4b9bb5f WatchSource:0}: Error finding container a97420c0c64a60bc1911e6552da271cd6f19a87abaebd4e0f101b30ca4b9bb5f: Status 404 returned error can't find the container with id a97420c0c64a60bc1911e6552da271cd6f19a87abaebd4e0f101b30ca4b9bb5f
Sep 08 13:28:14.411104 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.411085    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pl7gt"
Sep 08 13:28:14.414590 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.414569    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890f71c6_95de_42c3_9790_4d3c7155687d.slice/crio-f6ab3471a3586fd2e63ab1b19580d509f7e0396496f5846645b8a359d25bda7a WatchSource:0}: Error finding container f6ab3471a3586fd2e63ab1b19580d509f7e0396496f5846645b8a359d25bda7a: Status 404 returned error can't find the container with id f6ab3471a3586fd2e63ab1b19580d509f7e0396496f5846645b8a359d25bda7a
Sep 08 13:28:14.417514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.417482    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:28:14.418541 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:14.418520    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f67904d_8e19_44ee_a8ed_9da81a313102.slice/crio-938975d56248528336867790fec450d96286fdc36e9af05e00dc2eb50e4fad34 WatchSource:0}: Error finding container 938975d56248528336867790fec450d96286fdc36e9af05e00dc2eb50e4fad34: Status 404 returned error can't find the container with id 938975d56248528336867790fec450d96286fdc36e9af05e00dc2eb50e4fad34
Sep 08 13:28:14.682869 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.682781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:14.683039 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.682925    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:14.683039 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.682988    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:28:15.682969295 +0000 UTC m=+4.033955222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:14.783924 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.783888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:14.784102 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.784048    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:14.784151 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.784120    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret podName:04639862-2bd6-4c98-896e-473ccb2d7bd7 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:15.784103182 +0000 UTC m=+4.135089130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret") pod "global-pull-secret-syncer-wcdnb" (UID: "04639862-2bd6-4c98-896e-473ccb2d7bd7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:14.884610 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:14.884577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:14.884782 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.884704    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:28:14.884782 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.884720    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:28:14.884782 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.884730    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fgv2b for pod openshift-network-diagnostics/network-check-target-gf4f5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:14.884782 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:14.884780    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b podName:ad8bf542-ec0c-429c-8d95-40774e89f90c nodeName:}" failed. No retries permitted until 2026-09-08 13:28:15.884766275 +0000 UTC m=+4.235752202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgv2b" (UniqueName: "kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b") pod "network-check-target-gf4f5" (UID: "ad8bf542-ec0c-429c-8d95-40774e89f90c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:15.103757 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.103665    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:23:14 +0000 UTC" deadline="2028-05-24 12:58:14.823118546 +0000 UTC"
Sep 08 13:28:15.103757 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.103699    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14975h29m59.719425919s"
Sep 08 13:28:15.179250 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.179212    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:28:15.239918 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.239884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:15.240087 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.240024    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:15.250610 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.250572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bct9v" event={"ID":"902e01bc-c36e-4c70-be62-390a29c4697f","Type":"ContainerStarted","Data":"e0cc1969996023707ea5dce6bfa6d36b5dbcef9478c65053e0226c6bf2129be8"}
Sep 08 13:28:15.253533 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.253502    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vr5qx" event={"ID":"85e17bd3-c3e9-4b85-b945-e55797a49b90","Type":"ContainerStarted","Data":"80cb5e448e307b15a45c98a882b648ac015e1548239e53364edfc2c83d0d860d"}
Sep 08 13:28:15.255733 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.255707    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rjgp2" event={"ID":"10415704-0e6c-43d8-b16f-c5a2ee365b5e","Type":"ContainerStarted","Data":"6cb3ca568d7c5b66623813f96569655d78190f87bb2339b2b4368e8477fe2132"}
Sep 08 13:28:15.258250 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.258223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" event={"ID":"f3cc2402-36f3-4342-8d14-e978f9e66436","Type":"ContainerStarted","Data":"ea8696e16e1e89aa5af0a7fb9c78a33768602801b6788d6ec0468dfdd0c9381f"}
Sep 08 13:28:15.259990 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.259964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7z85" event={"ID":"38ef716f-8b3c-4b09-b816-1e59fdefc265","Type":"ContainerStarted","Data":"0cd7b4698694282d05c70f33de6092aa57aeb2ef60b3f942dc252dbd6d44db9f"}
Sep 08 13:28:15.262049 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.262024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pl7gt" event={"ID":"9f67904d-8e19-44ee-a8ed-9da81a313102","Type":"ContainerStarted","Data":"938975d56248528336867790fec450d96286fdc36e9af05e00dc2eb50e4fad34"}
Sep 08 13:28:15.263827 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.263803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"f6ab3471a3586fd2e63ab1b19580d509f7e0396496f5846645b8a359d25bda7a"}
Sep 08 13:28:15.268043 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.268017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt" event={"ID":"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4","Type":"ContainerStarted","Data":"a97420c0c64a60bc1911e6552da271cd6f19a87abaebd4e0f101b30ca4b9bb5f"}
Sep 08 13:28:15.271073 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.271048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hdfv6" event={"ID":"91b8aa20-62a5-4327-ba6f-563710623ce6","Type":"ContainerStarted","Data":"5d28bbdb2e05d8c769988fa6ccb225e969bf7ce415145d4d5ecc30c930360121"}
Sep 08 13:28:15.694743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.694695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:15.694946 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.694893    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:15.695218 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.695142    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:28:17.695115767 +0000 UTC m=+6.046101700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:15.796026 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.795991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:15.796187 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.796163    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:15.796239 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.796216    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret podName:04639862-2bd6-4c98-896e-473ccb2d7bd7 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:17.79620371 +0000 UTC m=+6.147189637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret") pod "global-pull-secret-syncer-wcdnb" (UID: "04639862-2bd6-4c98-896e-473ccb2d7bd7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:15.896828 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:15.896775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:15.897023 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.896964    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:28:15.897023 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.896985    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:28:15.897023 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.896998    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fgv2b for pod openshift-network-diagnostics/network-check-target-gf4f5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:15.897203 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:15.897060    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b podName:ad8bf542-ec0c-429c-8d95-40774e89f90c nodeName:}" failed. No retries permitted until 2026-09-08 13:28:17.897041544 +0000 UTC m=+6.248027498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgv2b" (UniqueName: "kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b") pod "network-check-target-gf4f5" (UID: "ad8bf542-ec0c-429c-8d95-40774e89f90c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:16.104855 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:16.104733    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:23:14 +0000 UTC" deadline="2028-06-15 18:45:58.30230923 +0000 UTC"
Sep 08 13:28:16.104855 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:16.104800    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15509h17m42.197513628s"
Sep 08 13:28:16.244802 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:16.244770    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:16.244966 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:16.244903    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:16.245475 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:16.245309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:16.245475 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:16.245424    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:17.239607 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:17.239535    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:17.240267 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.239663    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:17.712456 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:17.712361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:17.712615 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.712593    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:17.712672 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.712661    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:28:21.712641368 +0000 UTC m=+10.063627299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:17.813151 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:17.813040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:17.813333 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.813198    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:17.813333 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.813275    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret podName:04639862-2bd6-4c98-896e-473ccb2d7bd7 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:21.813259689 +0000 UTC m=+10.164245640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret") pod "global-pull-secret-syncer-wcdnb" (UID: "04639862-2bd6-4c98-896e-473ccb2d7bd7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:17.914489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:17.914446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:17.914750 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.914725    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:28:17.914750 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.914756    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:28:17.914960 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.914772    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fgv2b for pod openshift-network-diagnostics/network-check-target-gf4f5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:17.914960 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:17.914839    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b podName:ad8bf542-ec0c-429c-8d95-40774e89f90c nodeName:}" failed. No retries permitted until 2026-09-08 13:28:21.914818194 +0000 UTC m=+10.265804126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgv2b" (UniqueName: "kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b") pod "network-check-target-gf4f5" (UID: "ad8bf542-ec0c-429c-8d95-40774e89f90c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:18.239636 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:18.239595    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:18.240217 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:18.239823    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:18.240454 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:18.240361    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:18.240539 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:18.240496    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:19.240134 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:19.239603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:19.240134 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:19.239747    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:20.240087 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:20.240050    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:20.240277 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:20.240200    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:20.240707 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:20.240688    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:20.240815 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:20.240793    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:21.239199 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:21.239097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:21.239414 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.239237    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:21.680928 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:21.680175    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9135ca1286983be48fef1a9d4c6b86dc.slice/crio-b4747c032cdd374914b02c66869b7677a52884ef27b5567a62166051ab3f5649 WatchSource:0}: Error finding container b4747c032cdd374914b02c66869b7677a52884ef27b5567a62166051ab3f5649: Status 404 returned error can't find the container with id b4747c032cdd374914b02c66869b7677a52884ef27b5567a62166051ab3f5649
Sep 08 13:28:21.748496 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:21.748465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:21.748640 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.748622    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:21.748710 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.748692    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:28:29.748670766 +0000 UTC m=+18.099656707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:21.850061 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:21.849750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:21.850061 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.849915    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:21.850061 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.849975    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret podName:04639862-2bd6-4c98-896e-473ccb2d7bd7 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:29.849957101 +0000 UTC m=+18.200943033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret") pod "global-pull-secret-syncer-wcdnb" (UID: "04639862-2bd6-4c98-896e-473ccb2d7bd7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:21.950804 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:21.950725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:21.950956 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.950920    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:28:21.950956 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.950944    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:28:21.950956 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.950957    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fgv2b for pod openshift-network-diagnostics/network-check-target-gf4f5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:21.951116 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:21.951006    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b podName:ad8bf542-ec0c-429c-8d95-40774e89f90c nodeName:}" failed. No retries permitted until 2026-09-08 13:28:29.950995009 +0000 UTC m=+18.301980935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgv2b" (UniqueName: "kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b") pod "network-check-target-gf4f5" (UID: "ad8bf542-ec0c-429c-8d95-40774e89f90c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:22.240414 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.240083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:22.240414 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.240111    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:22.240588 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:22.240476    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:22.240922 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:22.240889    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:22.288770 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.288733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7z85" event={"ID":"38ef716f-8b3c-4b09-b816-1e59fdefc265","Type":"ContainerStarted","Data":"643699b502681058cf3bfd00d755b5bc910754a24f66503751b2c2328b692830"}
Sep 08 13:28:22.290110 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.290082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal" event={"ID":"9135ca1286983be48fef1a9d4c6b86dc","Type":"ContainerStarted","Data":"b4747c032cdd374914b02c66869b7677a52884ef27b5567a62166051ab3f5649"}
Sep 08 13:28:22.291681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.291660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pl7gt" event={"ID":"9f67904d-8e19-44ee-a8ed-9da81a313102","Type":"ContainerStarted","Data":"dc28a7f5b72b6727ed0e7065c3d935a13c5e77b3f26037006723202bf0fe56a0"}
Sep 08 13:28:22.293675 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.293650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt" event={"ID":"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4","Type":"ContainerStarted","Data":"99118dd4dcbe1bc460e85c4b9536b9ccab6f6c107c0251a9e05422009e745377"}
Sep 08 13:28:22.295250 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.295221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hdfv6" event={"ID":"91b8aa20-62a5-4327-ba6f-563710623ce6","Type":"ContainerStarted","Data":"aa2a1244e0a8b2d956f0e9978c8717ad4de8eb5bdb80dc55057df5b13887cbbf"}
Sep 08 13:28:22.296984 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.296960    2561 generic.go:356] "Generic (PLEG): container finished" podID="a1d821425d1dafec7de35f4d65d5b252" containerID="96db7716a049542297be6be758d5ef383c464801b60dd30f5a48c7d14c334263" exitCode=0
Sep 08 13:28:22.297082 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.297033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal" event={"ID":"a1d821425d1dafec7de35f4d65d5b252","Type":"ContainerDied","Data":"96db7716a049542297be6be758d5ef383c464801b60dd30f5a48c7d14c334263"}
Sep 08 13:28:22.299060 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.299034    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3cc2402-36f3-4342-8d14-e978f9e66436" containerID="cd52f0c2070290dd96220f5068ee40a369b1e84c8b374250cf7add14550df9ab" exitCode=0
Sep 08 13:28:22.299159 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.299072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" event={"ID":"f3cc2402-36f3-4342-8d14-e978f9e66436","Type":"ContainerDied","Data":"cd52f0c2070290dd96220f5068ee40a369b1e84c8b374250cf7add14550df9ab"}
Sep 08 13:28:22.304775 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.304728    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q7z85" podStartSLOduration=2.998341396 podStartE2EDuration="10.304710024s" totalImagesPullingTime="7.306368628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:22.304286074 +0000 UTC m=+10.655272023" watchObservedRunningTime="2026-09-08 13:28:22.304710024 +0000 UTC m=+10.655695973"
Sep 08 13:28:22.334547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.334483    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hdfv6" podStartSLOduration=3.844519921 podStartE2EDuration="10.33445947s" totalImagesPullingTime="6.489939549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:22.33413437 +0000 UTC m=+10.685120318" watchObservedRunningTime="2026-09-08 13:28:22.33445947 +0000 UTC m=+10.685445419"
Sep 08 13:28:22.368344 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:22.368285    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pl7gt" podStartSLOduration=2.115050129 podStartE2EDuration="9.368271385s" totalImagesPullingTime="7.253221256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:22.36773888 +0000 UTC m=+10.718724827" watchObservedRunningTime="2026-09-08 13:28:22.368271385 +0000 UTC m=+10.719257333"
Sep 08 13:28:23.239318 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:23.239287    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:23.239909 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:23.239435    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:24.239563 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:24.239529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:24.240041 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:24.239688    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:24.240041 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:24.239952    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:24.240144 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:24.240056    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:24.248874 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:24.248838    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:24.249624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:24.249602    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:24.306543 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:24.306510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vr5qx" event={"ID":"85e17bd3-c3e9-4b85-b945-e55797a49b90","Type":"ContainerStarted","Data":"6dd6f1f0d15e6b48227dcb40d1256d5af1f00b59ed4868d40a7725161e96fd7f"}
Sep 08 13:28:24.321585 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:24.321528    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vr5qx" podStartSLOduration=5.051870441 podStartE2EDuration="12.321512659s" totalImagesPullingTime="7.269642218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:24.321025833 +0000 UTC m=+12.672011781" watchObservedRunningTime="2026-09-08 13:28:24.321512659 +0000 UTC m=+12.672498607"
Sep 08 13:28:25.239803 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:25.239767    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:25.240299 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:25.239905    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:25.308388 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:25.308239    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:28:26.239732 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:26.239700    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:26.239946 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:26.239831    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:26.239946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:26.239860    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:26.240319 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:26.239961    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:27.239617 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:27.239570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:27.239807 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:27.239712    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:28.239555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:28.239082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:28.239555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:28.239082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:28.239555 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:28.239213    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:28.239555 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:28.239326    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:28.472983 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:28.472688    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:28.472983 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:28.472827    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:28:28.474023 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:28.473986    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q7z85"
Sep 08 13:28:29.239750 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:29.239721    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:29.240155 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:29.239831    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:29.812045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:29.811997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:29.812204 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:29.812149    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:29.812255 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:29.812215    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:28:45.812200472 +0000 UTC m=+34.163186404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:29.912693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:29.912655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:29.912891 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:29.912837    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:29.912972 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:29.912909    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret podName:04639862-2bd6-4c98-896e-473ccb2d7bd7 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:45.912890335 +0000 UTC m=+34.263876264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret") pod "global-pull-secret-syncer-wcdnb" (UID: "04639862-2bd6-4c98-896e-473ccb2d7bd7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:30.013859 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:30.013820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:30.014032 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:30.014007    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:28:30.014088 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:30.014036    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:28:30.014088 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:30.014049    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fgv2b for pod openshift-network-diagnostics/network-check-target-gf4f5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:30.014191 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:30.014116    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b podName:ad8bf542-ec0c-429c-8d95-40774e89f90c nodeName:}" failed. No retries permitted until 2026-09-08 13:28:46.014096424 +0000 UTC m=+34.365082367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgv2b" (UniqueName: "kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b") pod "network-check-target-gf4f5" (UID: "ad8bf542-ec0c-429c-8d95-40774e89f90c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:30.239914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:30.239832    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:30.240257 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:30.239832    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:30.240257 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:30.239968    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:30.240257 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:30.240031    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:31.239841 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:31.239804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:31.240023 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:31.239943    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:32.242630 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:32.242600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:32.243051 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:32.242605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:32.243051 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:32.242716    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:32.243051 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:32.242810    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:33.239877 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:33.239841    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:33.240059 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:33.239966    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:34.239507 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:34.239267    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:34.239942 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:34.239325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:34.239942 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:34.239628    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:34.239942 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:34.239718    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:34.328650 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:34.328591    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal" event={"ID":"a1d821425d1dafec7de35f4d65d5b252","Type":"ContainerStarted","Data":"a67694cd040b4634592c465544c41a57774c38629564f2199795397ee936729d"}
Sep 08 13:28:34.343614 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:34.343565    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-208.ec2.internal" podStartSLOduration=20.343549847 podStartE2EDuration="20.343549847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:34.342978132 +0000 UTC m=+22.693964079" watchObservedRunningTime="2026-09-08 13:28:34.343549847 +0000 UTC m=+22.694535794"
Sep 08 13:28:35.240028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:35.239986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:35.240441 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:35.240140    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:35.590598 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:35.590555    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:28:36.132663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.132339    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:28:35.59057798Z","UUID":"2557560b-5922-4657-a74a-563cdc52b876","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:28:36.134170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.134152    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:28:36.134170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.134183    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:28:36.239663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.239627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:36.239858 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:36.239766    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:36.239858 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.239812    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:36.239978 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:36.239891    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:36.334750 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.334712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rjgp2" event={"ID":"10415704-0e6c-43d8-b16f-c5a2ee365b5e","Type":"ContainerStarted","Data":"1890a05193c3db49da1e45a551d73635498abc4bb3b1e61fa8aa6687ce91c373"}
Sep 08 13:28:36.336481 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.336458    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3cc2402-36f3-4342-8d14-e978f9e66436" containerID="6ecb1d4168d5ed853415650394f287e7038761101d3287dea3ea31c111d0a176" exitCode=0
Sep 08 13:28:36.336588 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.336541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" event={"ID":"f3cc2402-36f3-4342-8d14-e978f9e66436","Type":"ContainerDied","Data":"6ecb1d4168d5ed853415650394f287e7038761101d3287dea3ea31c111d0a176"}
Sep 08 13:28:36.338141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.338114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal" event={"ID":"9135ca1286983be48fef1a9d4c6b86dc","Type":"ContainerStarted","Data":"6723ba101dc3aecb554646537c9c8ddf5c1375912f8648ebe28504c0674486a6"}
Sep 08 13:28:36.341196 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.341177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"f0d4d01bd4cdfe05458c7c0946371be258564307e7ff252ec1de23d2df5b7239"}
Sep 08 13:28:36.341319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.341234    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"a8b47e8c451d646b019edcb7004eb6b27248ac7e57218e7b8168ae273b8cc102"}
Sep 08 13:28:36.341319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.341264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"4556aa0aac1f085a4e5d6c5e29a9a0a4d4a113177fb765a89940805c132c26f0"}
Sep 08 13:28:36.341319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.341311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"9d30f880584ef221772ea29f5f31e0418ebf44b7b30e9e5edd0f03fa59aca1e8"}
Sep 08 13:28:36.341489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.341345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"0b6adec1d96f0a289bafc51e9abf119fc69e510334e4920eef296c744e7c9e7f"}
Sep 08 13:28:36.341489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.341372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"09a929199232a7d73254748da45bd87b2a0a21a36bd1d420f3adfe2b0cbed3fd"}
Sep 08 13:28:36.342881 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.342860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt" event={"ID":"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4","Type":"ContainerStarted","Data":"4b91a7914fbc85d0cf71e8579af39a53e62027adbad11d44b1d51a6ab98f77c6"}
Sep 08 13:28:36.344450 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.344430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bct9v" event={"ID":"902e01bc-c36e-4c70-be62-390a29c4697f","Type":"ContainerStarted","Data":"a676ba09efbce93bc9fde37aa62e2c84488e6c260256d03c1610871e4217e60e"}
Sep 08 13:28:36.352078 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.352038    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rjgp2" podStartSLOduration=3.577493317 podStartE2EDuration="24.352023586s" totalImagesPullingTime="20.774530269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:36.351607827 +0000 UTC m=+24.702593775" watchObservedRunningTime="2026-09-08 13:28:36.352023586 +0000 UTC m=+24.703009533"
Sep 08 13:28:36.367965 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:36.367915    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bct9v" podStartSLOduration=5.610648021 podStartE2EDuration="24.367899964s" totalImagesPullingTime="18.757251943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:36.367746457 +0000 UTC m=+24.718732405" watchObservedRunningTime="2026-09-08 13:28:36.367899964 +0000 UTC m=+24.718885912"
Sep 08 13:28:37.239934 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:37.239896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:37.240090 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:37.240002    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:38.239838 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:38.239757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:38.240203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:38.239758    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:38.240203 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:38.239850    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:38.240203 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:38.239940    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:38.353494 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:38.353461    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3cc2402-36f3-4342-8d14-e978f9e66436" containerID="8b8269a24ca96d6272ddd57147f3f58b22a43f0ca65ade363f7f7090d7b6bab4" exitCode=0
Sep 08 13:28:38.353661 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:38.353546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" event={"ID":"f3cc2402-36f3-4342-8d14-e978f9e66436","Type":"ContainerDied","Data":"8b8269a24ca96d6272ddd57147f3f58b22a43f0ca65ade363f7f7090d7b6bab4"}
Sep 08 13:28:38.357091 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:38.357072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"0b440329727c40863d681abd3071891c277ada3245e528f4ffc21000de334155"}
Sep 08 13:28:38.374538 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:38.374479    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-208.ec2.internal" podStartSLOduration=24.374464046 podStartE2EDuration="24.374464046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:36.407530832 +0000 UTC m=+24.758516780" watchObservedRunningTime="2026-09-08 13:28:38.374464046 +0000 UTC m=+26.725450031"
Sep 08 13:28:39.239401 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:39.239343    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:39.239580 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:39.239475    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:40.240141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.239925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:40.240899 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:40.240235    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:40.240899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.240016    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:40.240899 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:40.240688    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:40.363821 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.363794    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3cc2402-36f3-4342-8d14-e978f9e66436" containerID="8943b8868ed7164204756a08eb751e3f5533d8c51f409af922d9f58b10a203ae" exitCode=0
Sep 08 13:28:40.363922 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.363868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" event={"ID":"f3cc2402-36f3-4342-8d14-e978f9e66436","Type":"ContainerDied","Data":"8943b8868ed7164204756a08eb751e3f5533d8c51f409af922d9f58b10a203ae"}
Sep 08 13:28:40.368166 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.368143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" event={"ID":"890f71c6-95de-42c3-9790-4d3c7155687d","Type":"ContainerStarted","Data":"4e0ad791bb812ff592454791e1783bc20fc175a642077247bb291354960dd14c"}
Sep 08 13:28:40.371076 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.369925    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:40.371076 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.370051    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:40.371076 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.370113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:40.386132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.386107    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:40.387442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.387424    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:28:40.410438 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:40.410395    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc" podStartSLOduration=8.698164214 podStartE2EDuration="27.410359262s" totalImagesPullingTime="18.712195048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:40.409703637 +0000 UTC m=+28.760689594" watchObservedRunningTime="2026-09-08 13:28:40.410359262 +0000 UTC m=+28.761345210"
Sep 08 13:28:41.239597 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:41.239552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:41.239754 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:41.239681    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:41.374508 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:41.374021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt" event={"ID":"5462eacf-bac5-4bb8-9e6d-612d53bc8aa4","Type":"ContainerStarted","Data":"ac07e426cc34dd809e3b82d428d00ebff07ec91e35ade4eb464380346eb1c9f6"}
Sep 08 13:28:41.397333 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:41.397268    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp9wt" podStartSLOduration=4.045217386 podStartE2EDuration="29.397245194s" totalImagesPullingTime="25.352027808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:41.396525403 +0000 UTC m=+29.747511346" watchObservedRunningTime="2026-09-08 13:28:41.397245194 +0000 UTC m=+29.748231143"
Sep 08 13:28:42.116530 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:42.116492    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wcdnb"]
Sep 08 13:28:42.116772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:42.116612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:42.116772 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:42.116732    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:42.119747 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:42.119717    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zspg2"]
Sep 08 13:28:42.119911 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:42.119894    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:42.120081 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:42.120046    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:42.120368 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:42.120347    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gf4f5"]
Sep 08 13:28:42.120486 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:42.120453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:42.120553 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:42.120531    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:43.239698 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:43.239514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:43.240061 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:43.239772    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:44.239664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:44.239631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:44.239664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:44.239658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:44.240172 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:44.239769    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:44.240172 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:44.239823    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:45.239479 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:45.239451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:45.239642 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:45.239551    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:45.826693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:45.826606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:45.827052 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:45.826763    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:45.827052 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:45.826829    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:29:17.826813438 +0000 UTC m=+66.177799363 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:28:45.927440 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:45.927367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:45.927590 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:45.927500    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:45.927590 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:45.927564    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret podName:04639862-2bd6-4c98-896e-473ccb2d7bd7 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:17.927547189 +0000 UTC m=+66.278533114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret") pod "global-pull-secret-syncer-wcdnb" (UID: "04639862-2bd6-4c98-896e-473ccb2d7bd7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:28:46.028028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:46.027994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:46.028169 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:46.028150    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:28:46.028169 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:46.028169    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:28:46.028253 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:46.028179    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fgv2b for pod openshift-network-diagnostics/network-check-target-gf4f5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:46.028253 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:46.028228    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b podName:ad8bf542-ec0c-429c-8d95-40774e89f90c nodeName:}" failed. No retries permitted until 2026-09-08 13:29:18.0282153 +0000 UTC m=+66.379201231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgv2b" (UniqueName: "kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b") pod "network-check-target-gf4f5" (UID: "ad8bf542-ec0c-429c-8d95-40774e89f90c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:28:46.239625 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:46.239588    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:46.239770 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:46.239594    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:46.239770 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:46.239696    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wcdnb" podUID="04639862-2bd6-4c98-896e-473ccb2d7bd7"
Sep 08 13:28:46.239864 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:46.239763    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gf4f5" podUID="ad8bf542-ec0c-429c-8d95-40774e89f90c"
Sep 08 13:28:46.388148 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:46.388108    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3cc2402-36f3-4342-8d14-e978f9e66436" containerID="f2c1222aa5aecd7eec4a6c2341d6e975b2d5bc6aeac74e44a5487f37f54cbf43" exitCode=0
Sep 08 13:28:46.388148 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:46.388148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" event={"ID":"f3cc2402-36f3-4342-8d14-e978f9e66436","Type":"ContainerDied","Data":"f2c1222aa5aecd7eec4a6c2341d6e975b2d5bc6aeac74e44a5487f37f54cbf43"}
Sep 08 13:28:47.239365 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:47.239331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:47.239823 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:47.239464    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:28:47.393020 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:47.392988    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3cc2402-36f3-4342-8d14-e978f9e66436" containerID="e397ce5764b5fd2939d9cc84e55418b37c25cfa434e93b91d8ea98ebcf49ed48" exitCode=0
Sep 08 13:28:47.393194 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:47.393046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" event={"ID":"f3cc2402-36f3-4342-8d14-e978f9e66436","Type":"ContainerDied","Data":"e397ce5764b5fd2939d9cc84e55418b37c25cfa434e93b91d8ea98ebcf49ed48"}
Sep 08 13:28:47.983712 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:47.983537    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-208.ec2.internal" event="NodeReady"
Sep 08 13:28:47.983885 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:47.983801    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:28:48.014681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.014650    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57948c7d77-2m6cf"]
Sep 08 13:28:48.042102 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.042076    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z4pq8"]
Sep 08 13:28:48.042256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.042200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.045204 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.045178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 13:28:48.045589 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.045571    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 13:28:48.047161 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.047141    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9cf6z\""
Sep 08 13:28:48.051844 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.051824    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 13:28:48.053710 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.053693    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l2pgh"]
Sep 08 13:28:48.053808 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.053794    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.056739 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.056720    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:28:48.056868 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.056752    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jwgzv\""
Sep 08 13:28:48.056868 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.056718    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:28:48.058418 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.058400    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 13:28:48.069786 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.069761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57948c7d77-2m6cf"]
Sep 08 13:28:48.069786 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.069789    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4pq8"]
Sep 08 13:28:48.069931 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.069810    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l2pgh"]
Sep 08 13:28:48.069931 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.069875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l2pgh"
Sep 08 13:28:48.073177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.073152    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:28:48.073177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.073164    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rbspx\""
Sep 08 13:28:48.073177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.073169    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:28:48.073473 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.073306    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:28:48.142398 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-installation-pull-secrets\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.142398 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-registry-certificates\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.142576 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-bound-sa-token\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.142576 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2r6\" (UniqueName: \"kubernetes.io/projected/02d2b805-a009-4c2b-9400-9fedb9394010-kube-api-access-bz2r6\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.142576 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdsw\" (UniqueName: \"kubernetes.io/projected/5d9c002e-39e8-43b7-9548-f275c98d915e-kube-api-access-nvdsw\") pod \"ingress-canary-l2pgh\" (UID: \"5d9c002e-39e8-43b7-9548-f275c98d915e\") " pod="openshift-ingress-canary/ingress-canary-l2pgh"
Sep 08 13:28:48.142576 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-trusted-ca\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.142576 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02d2b805-a009-4c2b-9400-9fedb9394010-config-volume\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.142724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d9c002e-39e8-43b7-9548-f275c98d915e-cert\") pod \"ingress-canary-l2pgh\" (UID: \"5d9c002e-39e8-43b7-9548-f275c98d915e\") " pod="openshift-ingress-canary/ingress-canary-l2pgh"
Sep 08 13:28:48.142724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.142724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d2b805-a009-4c2b-9400-9fedb9394010-tmp-dir\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.142724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-image-registry-private-configuration\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.142724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnsm\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-kube-api-access-mdnsm\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.142724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.142892 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.142739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a9b257a-51d8-45a0-b821-87d616643748-ca-trust-extracted\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.239640 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.239605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:28:48.240003 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.239613    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:28:48.242863 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.242839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:28:48.243017 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.242839    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n94wj\""
Sep 08 13:28:48.243017 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.242839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:28:48.243017 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.242983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.243195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d2b805-a009-4c2b-9400-9fedb9394010-tmp-dir\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.243195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243030    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:28:48.243195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-image-registry-private-configuration\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.243195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnsm\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-kube-api-access-mdnsm\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.243195 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.243093    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:28:48.243195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.243195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243134    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a9b257a-51d8-45a0-b821-87d616643748-ca-trust-extracted\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.243195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-installation-pull-secrets\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.243108    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57948c7d77-2m6cf: secret "image-registry-tls" not found
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-registry-certificates\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-bound-sa-token\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.243265    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls podName:6a9b257a-51d8-45a0-b821-87d616643748 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:48.743242646 +0000 UTC m=+37.094228586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls") pod "image-registry-57948c7d77-2m6cf" (UID: "6a9b257a-51d8-45a0-b821-87d616643748") : secret "image-registry-tls" not found
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.243273    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2r6\" (UniqueName: \"kubernetes.io/projected/02d2b805-a009-4c2b-9400-9fedb9394010-kube-api-access-bz2r6\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.243333    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls podName:02d2b805-a009-4c2b-9400-9fedb9394010 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:48.743317042 +0000 UTC m=+37.094302975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls") pod "dns-default-z4pq8" (UID: "02d2b805-a009-4c2b-9400-9fedb9394010") : secret "dns-default-metrics-tls" not found
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d2b805-a009-4c2b-9400-9fedb9394010-tmp-dir\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdsw\" (UniqueName: \"kubernetes.io/projected/5d9c002e-39e8-43b7-9548-f275c98d915e-kube-api-access-nvdsw\") pod \"ingress-canary-l2pgh\" (UID: \"5d9c002e-39e8-43b7-9548-f275c98d915e\") " pod="openshift-ingress-canary/ingress-canary-l2pgh"
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-trusted-ca\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02d2b805-a009-4c2b-9400-9fedb9394010-config-volume\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.243755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d9c002e-39e8-43b7-9548-f275c98d915e-cert\") pod \"ingress-canary-l2pgh\" (UID: \"5d9c002e-39e8-43b7-9548-f275c98d915e\") " pod="openshift-ingress-canary/ingress-canary-l2pgh"
Sep 08 13:28:48.244314 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.243831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a9b257a-51d8-45a0-b821-87d616643748-ca-trust-extracted\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.244314 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.244169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-registry-certificates\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.244464 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.244441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02d2b805-a009-4c2b-9400-9fedb9394010-config-volume\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.244645 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.244624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-trusted-ca\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.247737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.247716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-installation-pull-secrets\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.247815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.247741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-image-registry-private-configuration\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.247815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.247776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d9c002e-39e8-43b7-9548-f275c98d915e-cert\") pod \"ingress-canary-l2pgh\" (UID: \"5d9c002e-39e8-43b7-9548-f275c98d915e\") " pod="openshift-ingress-canary/ingress-canary-l2pgh"
Sep 08 13:28:48.251271 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.251249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2r6\" (UniqueName: \"kubernetes.io/projected/02d2b805-a009-4c2b-9400-9fedb9394010-kube-api-access-bz2r6\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.251466 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.251441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-bound-sa-token\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.251571 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.251557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnsm\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-kube-api-access-mdnsm\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.252226 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.252211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdsw\" (UniqueName: \"kubernetes.io/projected/5d9c002e-39e8-43b7-9548-f275c98d915e-kube-api-access-nvdsw\") pod \"ingress-canary-l2pgh\" (UID: \"5d9c002e-39e8-43b7-9548-f275c98d915e\") " pod="openshift-ingress-canary/ingress-canary-l2pgh"
Sep 08 13:28:48.377923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.377884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l2pgh"
Sep 08 13:28:48.399121 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.399045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" event={"ID":"f3cc2402-36f3-4342-8d14-e978f9e66436","Type":"ContainerStarted","Data":"894362e295f049e6bb1a6f96c39133249268b12d80f395c9c312730074fc6204"}
Sep 08 13:28:48.421802 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.421746    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n8vsw" podStartSLOduration=8.572541409 podStartE2EDuration="36.421725969s" totalImagesPullingTime="27.84918456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:48.421081384 +0000 UTC m=+36.772067334" watchObservedRunningTime="2026-09-08 13:28:48.421725969 +0000 UTC m=+36.772711917"
Sep 08 13:28:48.697674 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.697637    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l2pgh"]
Sep 08 13:28:48.700944 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:48.700911    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9c002e_39e8_43b7_9548_f275c98d915e.slice/crio-1ecef503be9c722fef54bf5b789fcc9c8a8a58c8a7a6beb8de25033d66035164 WatchSource:0}: Error finding container 1ecef503be9c722fef54bf5b789fcc9c8a8a58c8a7a6beb8de25033d66035164: Status 404 returned error can't find the container with id 1ecef503be9c722fef54bf5b789fcc9c8a8a58c8a7a6beb8de25033d66035164
Sep 08 13:28:48.747461 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.747428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:48.747605 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:48.747466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:48.747605 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.747561    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:28:48.747605 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.747579    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:28:48.747605 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.747599    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57948c7d77-2m6cf: secret "image-registry-tls" not found
Sep 08 13:28:48.747740 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.747615    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls podName:02d2b805-a009-4c2b-9400-9fedb9394010 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:49.747599753 +0000 UTC m=+38.098585686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls") pod "dns-default-z4pq8" (UID: "02d2b805-a009-4c2b-9400-9fedb9394010") : secret "dns-default-metrics-tls" not found
Sep 08 13:28:48.747740 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:48.747656    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls podName:6a9b257a-51d8-45a0-b821-87d616643748 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:49.74764336 +0000 UTC m=+38.098629286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls") pod "image-registry-57948c7d77-2m6cf" (UID: "6a9b257a-51d8-45a0-b821-87d616643748") : secret "image-registry-tls" not found
Sep 08 13:28:49.239936 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:49.239904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:28:49.242723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:49.242703    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:28:49.242723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:49.242713    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-56h6n\""
Sep 08 13:28:49.402974 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:49.402938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l2pgh" event={"ID":"5d9c002e-39e8-43b7-9548-f275c98d915e","Type":"ContainerStarted","Data":"1ecef503be9c722fef54bf5b789fcc9c8a8a58c8a7a6beb8de25033d66035164"}
Sep 08 13:28:49.755365 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:49.755323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:49.755365 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:49.755366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:49.755594 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:49.755509    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:28:49.755594 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:49.755531    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57948c7d77-2m6cf: secret "image-registry-tls" not found
Sep 08 13:28:49.755594 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:49.755539    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:28:49.755594 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:49.755580    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls podName:02d2b805-a009-4c2b-9400-9fedb9394010 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:51.75556747 +0000 UTC m=+40.106553396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls") pod "dns-default-z4pq8" (UID: "02d2b805-a009-4c2b-9400-9fedb9394010") : secret "dns-default-metrics-tls" not found
Sep 08 13:28:49.755594 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:49.755593    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls podName:6a9b257a-51d8-45a0-b821-87d616643748 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:51.75558687 +0000 UTC m=+40.106572795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls") pod "image-registry-57948c7d77-2m6cf" (UID: "6a9b257a-51d8-45a0-b821-87d616643748") : secret "image-registry-tls" not found
Sep 08 13:28:50.289268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.289234    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"]
Sep 08 13:28:50.323005 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.322977    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"]
Sep 08 13:28:50.323170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.323122    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:28:50.325816 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.325786    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 13:28:50.325988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.325866    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 13:28:50.325988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.325919    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 13:28:50.325988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.325919    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 13:28:50.326138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.326074    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cr2mg\""
Sep 08 13:28:50.337056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.337034    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"]
Sep 08 13:28:50.337183 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.337084    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.339691 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.339671    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 13:28:50.353062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.353031    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"]
Sep 08 13:28:50.353062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.353058    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"]
Sep 08 13:28:50.353202 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.353078    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"]
Sep 08 13:28:50.353202 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.353144    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.355957 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.355938    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 13:28:50.356065 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.356018    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 13:28:50.356065 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.356027    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 13:28:50.356065 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.356036    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 13:28:50.461480 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19eeba2e-3be4-419a-a989-1eac46d73ae5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.461653 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9296\" (UniqueName: \"kubernetes.io/projected/59b7ae0f-2687-42b1-9f66-9e4632b47e7c-kube-api-access-s9296\") pod \"managed-serviceaccount-addon-agent-5c976c977d-rkqqs\" (UID: \"59b7ae0f-2687-42b1-9f66-9e4632b47e7c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:28:50.461653 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-ca\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.461723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45j6\" (UniqueName: \"kubernetes.io/projected/19eeba2e-3be4-419a-a989-1eac46d73ae5-kube-api-access-b45j6\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.461723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.461788 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.461788 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgv2c\" (UniqueName: \"kubernetes.io/projected/79709d16-5291-4c59-aab6-be8d72936b74-kube-api-access-mgv2c\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.461842 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19eeba2e-3be4-419a-a989-1eac46d73ae5-tmp\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.461842 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59b7ae0f-2687-42b1-9f66-9e4632b47e7c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c976c977d-rkqqs\" (UID: \"59b7ae0f-2687-42b1-9f66-9e4632b47e7c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:28:50.461842 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-hub\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.461842 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.461834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79709d16-5291-4c59-aab6-be8d72936b74-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.562433 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-ca\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.562433 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45j6\" (UniqueName: \"kubernetes.io/projected/19eeba2e-3be4-419a-a989-1eac46d73ae5-kube-api-access-b45j6\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.562433 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562425    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.562697 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.562697 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgv2c\" (UniqueName: \"kubernetes.io/projected/79709d16-5291-4c59-aab6-be8d72936b74-kube-api-access-mgv2c\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.562697 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19eeba2e-3be4-419a-a989-1eac46d73ae5-tmp\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.562879 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19eeba2e-3be4-419a-a989-1eac46d73ae5-tmp\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.562959 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59b7ae0f-2687-42b1-9f66-9e4632b47e7c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c976c977d-rkqqs\" (UID: \"59b7ae0f-2687-42b1-9f66-9e4632b47e7c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:28:50.562959 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-hub\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.563053 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.562971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79709d16-5291-4c59-aab6-be8d72936b74-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.563053 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.563011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19eeba2e-3be4-419a-a989-1eac46d73ae5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.563153 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.563059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9296\" (UniqueName: \"kubernetes.io/projected/59b7ae0f-2687-42b1-9f66-9e4632b47e7c-kube-api-access-s9296\") pod \"managed-serviceaccount-addon-agent-5c976c977d-rkqqs\" (UID: \"59b7ae0f-2687-42b1-9f66-9e4632b47e7c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:28:50.563987 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.563958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79709d16-5291-4c59-aab6-be8d72936b74-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.565255 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.565229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-ca\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.565600 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.565573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.565688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.565645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.565758 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.565702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/19eeba2e-3be4-419a-a989-1eac46d73ae5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.565758 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.565723    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79709d16-5291-4c59-aab6-be8d72936b74-hub\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.565824 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.565762    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59b7ae0f-2687-42b1-9f66-9e4632b47e7c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c976c977d-rkqqs\" (UID: \"59b7ae0f-2687-42b1-9f66-9e4632b47e7c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:28:50.570982 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.570955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9296\" (UniqueName: \"kubernetes.io/projected/59b7ae0f-2687-42b1-9f66-9e4632b47e7c-kube-api-access-s9296\") pod \"managed-serviceaccount-addon-agent-5c976c977d-rkqqs\" (UID: \"59b7ae0f-2687-42b1-9f66-9e4632b47e7c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:28:50.571872 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.571848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgv2c\" (UniqueName: \"kubernetes.io/projected/79709d16-5291-4c59-aab6-be8d72936b74-kube-api-access-mgv2c\") pod \"cluster-proxy-proxy-agent-b4d66b68b-jz6lg\" (UID: \"79709d16-5291-4c59-aab6-be8d72936b74\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.571959 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.571853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45j6\" (UniqueName: \"kubernetes.io/projected/19eeba2e-3be4-419a-a989-1eac46d73ae5-kube-api-access-b45j6\") pod \"klusterlet-addon-workmgr-5c8d8fbcc9-mp259\" (UID: \"19eeba2e-3be4-419a-a989-1eac46d73ae5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.643879 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.643844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:28:50.651021 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.650992    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:28:50.661150 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.661126    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:28:50.946166 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.945850    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"]
Sep 08 13:28:50.948601 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:50.948573    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59b7ae0f_2687_42b1_9f66_9e4632b47e7c.slice/crio-c7d5f32cbac2cf3879659a25c20aae3ac9f5da89c4882c0f91396b41f5f2947f WatchSource:0}: Error finding container c7d5f32cbac2cf3879659a25c20aae3ac9f5da89c4882c0f91396b41f5f2947f: Status 404 returned error can't find the container with id c7d5f32cbac2cf3879659a25c20aae3ac9f5da89c4882c0f91396b41f5f2947f
Sep 08 13:28:50.971854 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:50.971832    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"]
Sep 08 13:28:50.974179 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:50.974151    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19eeba2e_3be4_419a_a989_1eac46d73ae5.slice/crio-4ad163e65a7d57e3edcb3554f8aec0c1f799e36bb96cf7231a00358a81a1b957 WatchSource:0}: Error finding container 4ad163e65a7d57e3edcb3554f8aec0c1f799e36bb96cf7231a00358a81a1b957: Status 404 returned error can't find the container with id 4ad163e65a7d57e3edcb3554f8aec0c1f799e36bb96cf7231a00358a81a1b957
Sep 08 13:28:51.011661 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:51.011626    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"]
Sep 08 13:28:51.014911 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:28:51.014882    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79709d16_5291_4c59_aab6_be8d72936b74.slice/crio-73b658cf6cb6b8243f6ea58da022119547e238c09c9cc699dd71dec65d3518eb WatchSource:0}: Error finding container 73b658cf6cb6b8243f6ea58da022119547e238c09c9cc699dd71dec65d3518eb: Status 404 returned error can't find the container with id 73b658cf6cb6b8243f6ea58da022119547e238c09c9cc699dd71dec65d3518eb
Sep 08 13:28:51.409095 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:51.408990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259" event={"ID":"19eeba2e-3be4-419a-a989-1eac46d73ae5","Type":"ContainerStarted","Data":"4ad163e65a7d57e3edcb3554f8aec0c1f799e36bb96cf7231a00358a81a1b957"}
Sep 08 13:28:51.410131 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:51.410098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" event={"ID":"79709d16-5291-4c59-aab6-be8d72936b74","Type":"ContainerStarted","Data":"73b658cf6cb6b8243f6ea58da022119547e238c09c9cc699dd71dec65d3518eb"}
Sep 08 13:28:51.411077 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:51.411057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs" event={"ID":"59b7ae0f-2687-42b1-9f66-9e4632b47e7c","Type":"ContainerStarted","Data":"c7d5f32cbac2cf3879659a25c20aae3ac9f5da89c4882c0f91396b41f5f2947f"}
Sep 08 13:28:51.772549 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:51.771716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:51.772549 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:51.771783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:51.772549 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:51.771946    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:28:51.772549 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:51.772014    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls podName:02d2b805-a009-4c2b-9400-9fedb9394010 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:55.771995865 +0000 UTC m=+44.122981813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls") pod "dns-default-z4pq8" (UID: "02d2b805-a009-4c2b-9400-9fedb9394010") : secret "dns-default-metrics-tls" not found
Sep 08 13:28:51.772549 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:51.772444    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:28:51.772549 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:51.772461    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57948c7d77-2m6cf: secret "image-registry-tls" not found
Sep 08 13:28:51.772549 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:51.772508    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls podName:6a9b257a-51d8-45a0-b821-87d616643748 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:55.772493238 +0000 UTC m=+44.123479178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls") pod "image-registry-57948c7d77-2m6cf" (UID: "6a9b257a-51d8-45a0-b821-87d616643748") : secret "image-registry-tls" not found
Sep 08 13:28:55.807550 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:55.807511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:28:55.808069 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:28:55.807637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:28:55.808069 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:55.807664    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:28:55.808069 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:55.807730    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:28:55.808069 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:55.807742    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57948c7d77-2m6cf: secret "image-registry-tls" not found
Sep 08 13:28:55.808069 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:55.807765    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls podName:02d2b805-a009-4c2b-9400-9fedb9394010 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:03.807745669 +0000 UTC m=+52.158731606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls") pod "dns-default-z4pq8" (UID: "02d2b805-a009-4c2b-9400-9fedb9394010") : secret "dns-default-metrics-tls" not found
Sep 08 13:28:55.808069 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:28:55.807783    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls podName:6a9b257a-51d8-45a0-b821-87d616643748 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:03.80777307 +0000 UTC m=+52.158759010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls") pod "image-registry-57948c7d77-2m6cf" (UID: "6a9b257a-51d8-45a0-b821-87d616643748") : secret "image-registry-tls" not found
Sep 08 13:29:00.446981 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.446939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" event={"ID":"79709d16-5291-4c59-aab6-be8d72936b74","Type":"ContainerStarted","Data":"68fb0109dd1606b4d21b870ed92a72f9dd7c67f144a997351260642c3cd1c183"}
Sep 08 13:29:00.447400 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.446992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" event={"ID":"79709d16-5291-4c59-aab6-be8d72936b74","Type":"ContainerStarted","Data":"85eb2afd6d1b0dc8d3da8650a45c77dce9d1a032aab4854fc0b350d32c1fce0f"}
Sep 08 13:29:00.447400 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.447010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" event={"ID":"79709d16-5291-4c59-aab6-be8d72936b74","Type":"ContainerStarted","Data":"ffc1f2d6343fb01cdfd1620337ed00b1ffdbe313f49a308b5992f250dec9fd07"}
Sep 08 13:29:00.448423 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.448401    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l2pgh" event={"ID":"5d9c002e-39e8-43b7-9548-f275c98d915e","Type":"ContainerStarted","Data":"79b1c9c9e212c3db43d30f4917f6bcc264f97511ece0ebffa6b651b1d3c65dcc"}
Sep 08 13:29:00.449859 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.449830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs" event={"ID":"59b7ae0f-2687-42b1-9f66-9e4632b47e7c","Type":"ContainerStarted","Data":"30580ef6edd3b5fbe80a36d198e496adfe2dbdf308e9de5a772414e3bf73114c"}
Sep 08 13:29:00.451263 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.451240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259" event={"ID":"19eeba2e-3be4-419a-a989-1eac46d73ae5","Type":"ContainerStarted","Data":"cebe9eb7b81081639b13f24c7e7e3657c9b097425ee8080376b6096e8e2deeab"}
Sep 08 13:29:00.451457 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.451443    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:29:00.453140 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.453124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:29:00.464552 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.464512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" podStartSLOduration=1.980796444 podStartE2EDuration="10.464500067s" totalImagesPullingTime="8.483703623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:00.464127171 +0000 UTC m=+48.815113120" watchObservedRunningTime="2026-09-08 13:29:00.464500067 +0000 UTC m=+48.815486015"
Sep 08 13:29:00.479467 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.479424    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs" podStartSLOduration=1.944979963 podStartE2EDuration="10.47941131s" totalImagesPullingTime="8.534431347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:00.478954794 +0000 UTC m=+48.829940742" watchObservedRunningTime="2026-09-08 13:29:00.47941131 +0000 UTC m=+48.830397258"
Sep 08 13:29:00.495804 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.495764    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l2pgh" podStartSLOduration=1.71273356 podStartE2EDuration="12.495750857s" totalImagesPullingTime="10.783017297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:00.495604883 +0000 UTC m=+48.846590832" watchObservedRunningTime="2026-09-08 13:29:00.495750857 +0000 UTC m=+48.846736805"
Sep 08 13:29:00.511659 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:00.511603    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259" podStartSLOduration=2.002151342 podStartE2EDuration="10.51158228s" totalImagesPullingTime="8.509430938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:00.51055845 +0000 UTC m=+48.861544399" watchObservedRunningTime="2026-09-08 13:29:00.51158228 +0000 UTC m=+48.862568229"
Sep 08 13:29:03.877360 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:03.877268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:29:03.877360 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:03.877314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:29:03.877803 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:03.877424    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:29:03.877803 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:03.877445    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57948c7d77-2m6cf: secret "image-registry-tls" not found
Sep 08 13:29:03.877803 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:03.877495    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls podName:6a9b257a-51d8-45a0-b821-87d616643748 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:19.877480263 +0000 UTC m=+68.228466189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls") pod "image-registry-57948c7d77-2m6cf" (UID: "6a9b257a-51d8-45a0-b821-87d616643748") : secret "image-registry-tls" not found
Sep 08 13:29:03.877803 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:03.877445    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:29:03.877803 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:03.877527    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls podName:02d2b805-a009-4c2b-9400-9fedb9394010 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:19.877520822 +0000 UTC m=+68.228506748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls") pod "dns-default-z4pq8" (UID: "02d2b805-a009-4c2b-9400-9fedb9394010") : secret "dns-default-metrics-tls" not found
Sep 08 13:29:12.387158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:12.387131    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bjbc"
Sep 08 13:29:17.881043 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:17.880992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:29:17.884010 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:17.883990    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:29:17.892194 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:17.892173    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:29:17.892292 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:17.892270    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:30:21.892247715 +0000 UTC m=+130.243233640 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : secret "metrics-daemon-secret" not found
Sep 08 13:29:17.982162 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:17.982129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:29:17.985161 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:17.985145    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:29:17.996057 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:17.996029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04639862-2bd6-4c98-896e-473ccb2d7bd7-original-pull-secret\") pod \"global-pull-secret-syncer-wcdnb\" (UID: \"04639862-2bd6-4c98-896e-473ccb2d7bd7\") " pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:29:18.083118 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.083080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:29:18.085992 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.085973    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:29:18.095760 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.095740    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:29:18.106736 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.106714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgv2b\" (UniqueName: \"kubernetes.io/projected/ad8bf542-ec0c-429c-8d95-40774e89f90c-kube-api-access-fgv2b\") pod \"network-check-target-gf4f5\" (UID: \"ad8bf542-ec0c-429c-8d95-40774e89f90c\") " pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:29:18.262874 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.262838    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wcdnb"
Sep 08 13:29:18.270126 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.270101    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n94wj\""
Sep 08 13:29:18.277430 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.277403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:29:18.536825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.536737    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wcdnb"]
Sep 08 13:29:18.540248 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:29:18.540212    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04639862_2bd6_4c98_896e_473ccb2d7bd7.slice/crio-6a5e61ce79043ed21f39d4f84ef09dd1f501ab60d32bc49235cf256ff3c7df25 WatchSource:0}: Error finding container 6a5e61ce79043ed21f39d4f84ef09dd1f501ab60d32bc49235cf256ff3c7df25: Status 404 returned error can't find the container with id 6a5e61ce79043ed21f39d4f84ef09dd1f501ab60d32bc49235cf256ff3c7df25
Sep 08 13:29:18.558079 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:18.558054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gf4f5"]
Sep 08 13:29:18.561050 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:29:18.561028    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8bf542_ec0c_429c_8d95_40774e89f90c.slice/crio-1a9682eac09a1f055943a93a4fd555e2c36088311972e3366e3f7c162e39b8f5 WatchSource:0}: Error finding container 1a9682eac09a1f055943a93a4fd555e2c36088311972e3366e3f7c162e39b8f5: Status 404 returned error can't find the container with id 1a9682eac09a1f055943a93a4fd555e2c36088311972e3366e3f7c162e39b8f5
Sep 08 13:29:19.510925 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:19.510891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wcdnb" event={"ID":"04639862-2bd6-4c98-896e-473ccb2d7bd7","Type":"ContainerStarted","Data":"6a5e61ce79043ed21f39d4f84ef09dd1f501ab60d32bc49235cf256ff3c7df25"}
Sep 08 13:29:19.511928 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:19.511905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gf4f5" event={"ID":"ad8bf542-ec0c-429c-8d95-40774e89f90c","Type":"ContainerStarted","Data":"1a9682eac09a1f055943a93a4fd555e2c36088311972e3366e3f7c162e39b8f5"}
Sep 08 13:29:19.895920 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:19.895821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:29:19.895920 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:19.895868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:29:19.896154 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:19.895971    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:29:19.896154 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:19.895978    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:29:19.896154 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:19.896001    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57948c7d77-2m6cf: secret "image-registry-tls" not found
Sep 08 13:29:19.896154 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:19.896020    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls podName:02d2b805-a009-4c2b-9400-9fedb9394010 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:51.896008274 +0000 UTC m=+100.246994199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls") pod "dns-default-z4pq8" (UID: "02d2b805-a009-4c2b-9400-9fedb9394010") : secret "dns-default-metrics-tls" not found
Sep 08 13:29:19.896154 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:19.896045    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls podName:6a9b257a-51d8-45a0-b821-87d616643748 nodeName:}" failed. No retries permitted until 2026-09-08 13:29:51.896031445 +0000 UTC m=+100.247017391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls") pod "image-registry-57948c7d77-2m6cf" (UID: "6a9b257a-51d8-45a0-b821-87d616643748") : secret "image-registry-tls" not found
Sep 08 13:29:26.536845 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:26.536807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wcdnb" event={"ID":"04639862-2bd6-4c98-896e-473ccb2d7bd7","Type":"ContainerStarted","Data":"03296f259b79317d7c20663bc10dedb74ddda7c264a3870a60104718ef597975"}
Sep 08 13:29:26.551479 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:26.551426    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wcdnb" podStartSLOduration=66.483366195 podStartE2EDuration="1m13.551411394s" totalImagesPullingTime="7.068045199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:26.551172 +0000 UTC m=+74.902157950" watchObservedRunningTime="2026-09-08 13:29:26.551411394 +0000 UTC m=+74.902397342"
Sep 08 13:29:29.548000 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:29.547915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gf4f5" event={"ID":"ad8bf542-ec0c-429c-8d95-40774e89f90c","Type":"ContainerStarted","Data":"cd8e15b0877d5fa7a98ae3e13358198773437781e14d0da5d8724858c71afbf7"}
Sep 08 13:29:29.548370 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:29.548045    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:29:29.565727 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:29.565683    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gf4f5" podStartSLOduration=65.898361771 podStartE2EDuration="1m16.565663974s" totalImagesPullingTime="10.667302203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:29.565457251 +0000 UTC m=+77.916443200" watchObservedRunningTime="2026-09-08 13:29:29.565663974 +0000 UTC m=+77.916649939"
Sep 08 13:29:51.944494 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:51.944444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:29:51.945037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:29:51.944566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:29:51.945037 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:51.944623    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:29:51.945037 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:51.944642    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57948c7d77-2m6cf: secret "image-registry-tls" not found
Sep 08 13:29:51.945037 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:51.944733    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls podName:6a9b257a-51d8-45a0-b821-87d616643748 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:55.944711864 +0000 UTC m=+164.295697810 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls") pod "image-registry-57948c7d77-2m6cf" (UID: "6a9b257a-51d8-45a0-b821-87d616643748") : secret "image-registry-tls" not found
Sep 08 13:29:51.945037 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:51.944818    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:29:51.945280 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:29:51.945068    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls podName:02d2b805-a009-4c2b-9400-9fedb9394010 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:55.945043057 +0000 UTC m=+164.296028999 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls") pod "dns-default-z4pq8" (UID: "02d2b805-a009-4c2b-9400-9fedb9394010") : secret "dns-default-metrics-tls" not found
Sep 08 13:30:00.553124 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:00.553084    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gf4f5"
Sep 08 13:30:21.964869 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:21.964825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:30:21.965284 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:21.964985    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:30:21.965284 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:21.965066    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs podName:6404c150-528e-4903-8049-9fa576f8f9ea nodeName:}" failed. No retries permitted until 2026-09-08 13:32:23.965047505 +0000 UTC m=+252.316033431 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs") pod "network-metrics-daemon-zspg2" (UID: "6404c150-528e-4903-8049-9fa576f8f9ea") : secret "metrics-daemon-secret" not found
Sep 08 13:30:25.474840 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.474806    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zzgcc"]
Sep 08 13:30:25.476833 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.476813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.479923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.479893    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j78zr\""
Sep 08 13:30:25.479923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.479904    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:30:25.480841 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.480815    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:30:25.480841 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.480832    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:30:25.481031 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.480906    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:30:25.493693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.490532    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zzgcc"]
Sep 08 13:30:25.595719 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.595683    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c27f9499-cc5a-47ec-849f-838cd3698568-crio-socket\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.595719 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.595718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c27f9499-cc5a-47ec-849f-838cd3698568-data-volume\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.595923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.595749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.595923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.595775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4dhx\" (UniqueName: \"kubernetes.io/projected/c27f9499-cc5a-47ec-849f-838cd3698568-kube-api-access-r4dhx\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.595923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.595833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.697190 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.697149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c27f9499-cc5a-47ec-849f-838cd3698568-crio-socket\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.697190 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.697186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c27f9499-cc5a-47ec-849f-838cd3698568-data-volume\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.697471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.697212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.697471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.697229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4dhx\" (UniqueName: \"kubernetes.io/projected/c27f9499-cc5a-47ec-849f-838cd3698568-kube-api-access-r4dhx\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.697471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.697257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.697471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.697368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c27f9499-cc5a-47ec-849f-838cd3698568-crio-socket\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.697471 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:25.697447    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:30:25.697652 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:25.697482    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:25.697652 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:25.697525    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:26.19750822 +0000 UTC m=+134.548494146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : configmap "kube-rbac-proxy" not found
Sep 08 13:30:25.697652 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:25.697557    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:26.197538271 +0000 UTC m=+134.548524202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:25.697652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.697590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c27f9499-cc5a-47ec-849f-838cd3698568-data-volume\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:25.705924 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:25.705895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4dhx\" (UniqueName: \"kubernetes.io/projected/c27f9499-cc5a-47ec-849f-838cd3698568-kube-api-access-r4dhx\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:26.200441 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:26.200363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:26.200652 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:26.200510    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:26.200652 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:26.200570    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:30:26.200652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:26.200518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:26.200652 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:26.200573    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:27.200558564 +0000 UTC m=+135.551544509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:26.200652 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:26.200635    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:27.200618633 +0000 UTC m=+135.551604562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : configmap "kube-rbac-proxy" not found
Sep 08 13:30:27.208506 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:27.208467    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:27.208960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:27.208534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:27.208960 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:27.208603    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:30:27.208960 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:27.208667    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:27.208960 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:27.208681    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:29.208663887 +0000 UTC m=+137.559649816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : configmap "kube-rbac-proxy" not found
Sep 08 13:30:27.208960 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:27.208719    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:29.208703485 +0000 UTC m=+137.559689422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:27.440496 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:27.440470    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pl7gt_9f67904d-8e19-44ee-a8ed-9da81a313102/dns-node-resolver/0.log"
Sep 08 13:30:28.441461 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:28.441435    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hdfv6_91b8aa20-62a5-4327-ba6f-563710623ce6/node-ca/0.log"
Sep 08 13:30:29.223325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:29.223280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:29.223325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:29.223341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:29.223567 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:29.223444    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:30:29.223567 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:29.223482    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:29.223567 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:29.223510    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:33.223494826 +0000 UTC m=+141.574480751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : configmap "kube-rbac-proxy" not found
Sep 08 13:30:29.223567 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:29.223525    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:33.22351899 +0000 UTC m=+141.574504915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:29.240406 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:29.240370    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l2pgh_5d9c002e-39e8-43b7-9548-f275c98d915e/serve-healthcheck-canary/0.log"
Sep 08 13:30:33.256158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:33.256096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:33.256648 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:33.256192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:33.256648 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:33.256232    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:30:33.256648 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:33.256320    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:41.256299837 +0000 UTC m=+149.607285856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : configmap "kube-rbac-proxy" not found
Sep 08 13:30:33.256648 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:33.256346    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:33.256648 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:33.256433    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls podName:c27f9499-cc5a-47ec-849f-838cd3698568 nodeName:}" failed. No retries permitted until 2026-09-08 13:30:41.256418892 +0000 UTC m=+149.607404823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zzgcc" (UID: "c27f9499-cc5a-47ec-849f-838cd3698568") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:30:41.325327 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:41.325282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:41.325793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:41.325445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:41.325841 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:41.325815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c27f9499-cc5a-47ec-849f-838cd3698568-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:41.327658 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:41.327637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c27f9499-cc5a-47ec-849f-838cd3698568-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zzgcc\" (UID: \"c27f9499-cc5a-47ec-849f-838cd3698568\") " pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:41.385968 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:41.385936    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zzgcc"
Sep 08 13:30:41.651158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:41.651132    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zzgcc"]
Sep 08 13:30:41.653856 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:30:41.653828    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27f9499_cc5a_47ec_849f_838cd3698568.slice/crio-25ecdad9f2169331c69efab141bd808dd1ec244ce025255750b4d89107114d34 WatchSource:0}: Error finding container 25ecdad9f2169331c69efab141bd808dd1ec244ce025255750b4d89107114d34: Status 404 returned error can't find the container with id 25ecdad9f2169331c69efab141bd808dd1ec244ce025255750b4d89107114d34
Sep 08 13:30:41.761247 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:41.761209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zzgcc" event={"ID":"c27f9499-cc5a-47ec-849f-838cd3698568","Type":"ContainerStarted","Data":"297152de77ca7ceeee39b835501ab196790b922c7de0ae58a1cb46e73b8f1198"}
Sep 08 13:30:41.761429 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:41.761267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zzgcc" event={"ID":"c27f9499-cc5a-47ec-849f-838cd3698568","Type":"ContainerStarted","Data":"25ecdad9f2169331c69efab141bd808dd1ec244ce025255750b4d89107114d34"}
Sep 08 13:30:44.771853 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:44.771816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zzgcc" event={"ID":"c27f9499-cc5a-47ec-849f-838cd3698568","Type":"ContainerStarted","Data":"68e9340bf21a1646ae3f9d58bbc89cd857918d9033007e0d57561f5ff24071bc"}
Sep 08 13:30:48.785688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:48.785646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zzgcc" event={"ID":"c27f9499-cc5a-47ec-849f-838cd3698568","Type":"ContainerStarted","Data":"964bf19e6c39d7ce29fc4fe1d9a21c04f65314a836cdc56c5c81d2744c2fa789"}
Sep 08 13:30:48.818123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:48.818070    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zzgcc" podStartSLOduration=17.604234898 podStartE2EDuration="23.818053842s" totalImagesPullingTime="6.213818944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:30:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:48.817000017 +0000 UTC m=+157.167985964" watchObservedRunningTime="2026-09-08 13:30:48.818053842 +0000 UTC m=+157.169039822"
Sep 08 13:30:51.052992 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:51.052945    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf" podUID="6a9b257a-51d8-45a0-b821-87d616643748"
Sep 08 13:30:51.063140 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:51.063104    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z4pq8" podUID="02d2b805-a009-4c2b-9400-9fedb9394010"
Sep 08 13:30:51.795746 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:51.795713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:30:51.795920 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:51.795713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:30:52.248845 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:30:52.248806    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zspg2" podUID="6404c150-528e-4903-8049-9fa576f8f9ea"
Sep 08 13:30:56.035284 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.035226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:30:56.035284 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.035284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:30:56.037684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.037648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d2b805-a009-4c2b-9400-9fedb9394010-metrics-tls\") pod \"dns-default-z4pq8\" (UID: \"02d2b805-a009-4c2b-9400-9fedb9394010\") " pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:30:56.037786 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.037742    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"image-registry-57948c7d77-2m6cf\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") " pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:30:56.300360 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.300269    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9cf6z\""
Sep 08 13:30:56.300360 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.300269    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jwgzv\""
Sep 08 13:30:56.307398 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.307360    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:30:56.307539 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.307479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:30:56.586012 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.585930    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4pq8"]
Sep 08 13:30:56.588934 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:30:56.588889    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d2b805_a009_4c2b_9400_9fedb9394010.slice/crio-82a8d87c371ff185af7f69b8f87d582e978a3121d02aec0bc8b59253c3a10125 WatchSource:0}: Error finding container 82a8d87c371ff185af7f69b8f87d582e978a3121d02aec0bc8b59253c3a10125: Status 404 returned error can't find the container with id 82a8d87c371ff185af7f69b8f87d582e978a3121d02aec0bc8b59253c3a10125
Sep 08 13:30:56.589838 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.589724    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57948c7d77-2m6cf"]
Sep 08 13:30:56.601678 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:30:56.601653    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9b257a_51d8_45a0_b821_87d616643748.slice/crio-b39ceaa7900c8ef196134b28813d7b60cfbf63018cbc4c53e0b39707f7beabdc WatchSource:0}: Error finding container b39ceaa7900c8ef196134b28813d7b60cfbf63018cbc4c53e0b39707f7beabdc: Status 404 returned error can't find the container with id b39ceaa7900c8ef196134b28813d7b60cfbf63018cbc4c53e0b39707f7beabdc
Sep 08 13:30:56.812522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.812478    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf" event={"ID":"6a9b257a-51d8-45a0-b821-87d616643748","Type":"ContainerStarted","Data":"ea27efd53ed54bce9bba4f0a5de5709c875f1f47c8a1e57b0ddfb4b50233b305"}
Sep 08 13:30:56.812721 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.812581    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf" event={"ID":"6a9b257a-51d8-45a0-b821-87d616643748","Type":"ContainerStarted","Data":"b39ceaa7900c8ef196134b28813d7b60cfbf63018cbc4c53e0b39707f7beabdc"}
Sep 08 13:30:56.812721 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.812645    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:30:56.813721 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.813695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4pq8" event={"ID":"02d2b805-a009-4c2b-9400-9fedb9394010","Type":"ContainerStarted","Data":"82a8d87c371ff185af7f69b8f87d582e978a3121d02aec0bc8b59253c3a10125"}
Sep 08 13:30:56.831286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:56.831234    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf" podStartSLOduration=163.831218076 podStartE2EDuration="2m43.831218076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:56.829647111 +0000 UTC m=+165.180633060" watchObservedRunningTime="2026-09-08 13:30:56.831218076 +0000 UTC m=+165.182204068"
Sep 08 13:30:58.823284 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:58.823255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4pq8" event={"ID":"02d2b805-a009-4c2b-9400-9fedb9394010","Type":"ContainerStarted","Data":"1e60c361193c2a001f1384fb3eb94dabbf6f25caba19aa5c26d3bc8b394d9f74"}
Sep 08 13:30:59.828859 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.828827    2561 generic.go:356] "Generic (PLEG): container finished" podID="59b7ae0f-2687-42b1-9f66-9e4632b47e7c" containerID="30580ef6edd3b5fbe80a36d198e496adfe2dbdf308e9de5a772414e3bf73114c" exitCode=255
Sep 08 13:30:59.829290 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.828904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs" event={"ID":"59b7ae0f-2687-42b1-9f66-9e4632b47e7c","Type":"ContainerDied","Data":"30580ef6edd3b5fbe80a36d198e496adfe2dbdf308e9de5a772414e3bf73114c"}
Sep 08 13:30:59.829290 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.829250    2561 scope.go:117] "RemoveContainer" containerID="30580ef6edd3b5fbe80a36d198e496adfe2dbdf308e9de5a772414e3bf73114c"
Sep 08 13:30:59.830966 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.830944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4pq8" event={"ID":"02d2b805-a009-4c2b-9400-9fedb9394010","Type":"ContainerStarted","Data":"5d21a7eb91ad2ae17b96ecba3a10a86a38e2877ee43faa55bc5c398e4eeb601e"}
Sep 08 13:30:59.831065 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.830996    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:30:59.832480 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.832455    2561 generic.go:356] "Generic (PLEG): container finished" podID="19eeba2e-3be4-419a-a989-1eac46d73ae5" containerID="cebe9eb7b81081639b13f24c7e7e3657c9b097425ee8080376b6096e8e2deeab" exitCode=1
Sep 08 13:30:59.832568 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.832495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259" event={"ID":"19eeba2e-3be4-419a-a989-1eac46d73ae5","Type":"ContainerDied","Data":"cebe9eb7b81081639b13f24c7e7e3657c9b097425ee8080376b6096e8e2deeab"}
Sep 08 13:30:59.832772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.832760    2561 scope.go:117] "RemoveContainer" containerID="cebe9eb7b81081639b13f24c7e7e3657c9b097425ee8080376b6096e8e2deeab"
Sep 08 13:30:59.879918 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:30:59.879831    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z4pq8" podStartSLOduration=129.735897028 podStartE2EDuration="2m11.87981628s" totalImagesPullingTime="2.143919252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:59.861576433 +0000 UTC m=+168.212562383" watchObservedRunningTime="2026-09-08 13:30:59.87981628 +0000 UTC m=+168.230802228"
Sep 08 13:31:00.452119 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.452080    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:31:00.644366 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.644334    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs"
Sep 08 13:31:00.651463 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.651438    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:31:00.838048 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.837959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c976c977d-rkqqs" event={"ID":"59b7ae0f-2687-42b1-9f66-9e4632b47e7c","Type":"ContainerStarted","Data":"ac66c63ea4ee18cea40768c594d5fbedf9f068e5ab1235337ad0d441927c8385"}
Sep 08 13:31:00.839871 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.839845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259" event={"ID":"19eeba2e-3be4-419a-a989-1eac46d73ae5","Type":"ContainerStarted","Data":"ff30b9919b18b5b1c67e0f6943c382687c57a555ad1a1620dd49ed7cf3fc272c"}
Sep 08 13:31:00.899800 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.899765    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-prjvg"]
Sep 08 13:31:00.902001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.901978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:00.904423 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.904404    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:31:00.904785 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.904768    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:31:00.904923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.904903    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:31:00.905044 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.905028    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:31:00.905119 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.905050    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:31:00.905119 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.905096    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:31:00.905959 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:00.905938    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lm4lr\""
Sep 08 13:31:01.072262 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-tls\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.072481 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-textfile\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.072481 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc18798d-e11e-4b96-8e80-14e22e838765-metrics-client-ca\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.072481 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.072481 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-wtmp\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.072481 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-accelerators-collector-config\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.072664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxr6n\" (UniqueName: \"kubernetes.io/projected/bc18798d-e11e-4b96-8e80-14e22e838765-kube-api-access-zxr6n\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.072664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-root\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.072664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.072541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-sys\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.172982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-accelerators-collector-config\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxr6n\" (UniqueName: \"kubernetes.io/projected/bc18798d-e11e-4b96-8e80-14e22e838765-kube-api-access-zxr6n\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-root\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-sys\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-tls\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-textfile\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc18798d-e11e-4b96-8e80-14e22e838765-metrics-client-ca\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-sys\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173587 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-root\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173587 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-wtmp\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173587 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-wtmp\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173587 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-textfile\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173822 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc18798d-e11e-4b96-8e80-14e22e838765-metrics-client-ca\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.173945 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.173851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-accelerators-collector-config\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.175606 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.175570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-tls\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.175606 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.175589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc18798d-e11e-4b96-8e80-14e22e838765-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.181107 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.181088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxr6n\" (UniqueName: \"kubernetes.io/projected/bc18798d-e11e-4b96-8e80-14e22e838765-kube-api-access-zxr6n\") pod \"node-exporter-prjvg\" (UID: \"bc18798d-e11e-4b96-8e80-14e22e838765\") " pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.211202 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.211171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prjvg"
Sep 08 13:31:01.220712 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:31:01.220680    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc18798d_e11e_4b96_8e80_14e22e838765.slice/crio-2759ed725d961e11ce3352cb0a0dfa4b0d889dc507390c0c40a38287cdf83980 WatchSource:0}: Error finding container 2759ed725d961e11ce3352cb0a0dfa4b0d889dc507390c0c40a38287cdf83980: Status 404 returned error can't find the container with id 2759ed725d961e11ce3352cb0a0dfa4b0d889dc507390c0c40a38287cdf83980
Sep 08 13:31:01.844429 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.844370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prjvg" event={"ID":"bc18798d-e11e-4b96-8e80-14e22e838765","Type":"ContainerStarted","Data":"2759ed725d961e11ce3352cb0a0dfa4b0d889dc507390c0c40a38287cdf83980"}
Sep 08 13:31:01.844875 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.844623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:31:01.846042 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:01.846020    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c8d8fbcc9-mp259"
Sep 08 13:31:05.859482 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:05.859443    2561 generic.go:356] "Generic (PLEG): container finished" podID="bc18798d-e11e-4b96-8e80-14e22e838765" containerID="e08345bb1bacb72d9c6228b7dc5f028a47f6bf06b0cdca7fe8aa62e731ce842d" exitCode=0
Sep 08 13:31:05.859913 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:05.859530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prjvg" event={"ID":"bc18798d-e11e-4b96-8e80-14e22e838765","Type":"ContainerDied","Data":"e08345bb1bacb72d9c6228b7dc5f028a47f6bf06b0cdca7fe8aa62e731ce842d"}
Sep 08 13:31:06.864755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:06.864719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prjvg" event={"ID":"bc18798d-e11e-4b96-8e80-14e22e838765","Type":"ContainerStarted","Data":"cfe2dd905474921d3a8f639f87ca64ec294fb183575b77d154faf140c9deb1eb"}
Sep 08 13:31:06.865150 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:06.864772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prjvg" event={"ID":"bc18798d-e11e-4b96-8e80-14e22e838765","Type":"ContainerStarted","Data":"3a1a8c3f89ed6ca5086a696b04d315fbd9c34d790b87045783afbb192a7f224d"}
Sep 08 13:31:06.885918 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:06.885871    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-prjvg" podStartSLOduration=2.72973649 podStartE2EDuration="6.88585408s" totalImagesPullingTime="4.093559537s" totalInitContainerRuntime="62.558053ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:31:06.883959788 +0000 UTC m=+175.234945738" watchObservedRunningTime="2026-09-08 13:31:06.88585408 +0000 UTC m=+175.236840027"
Sep 08 13:31:08.239276 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:08.239242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:31:09.842664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:09.842628    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z4pq8"
Sep 08 13:31:11.801451 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:11.801412    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57948c7d77-2m6cf"]
Sep 08 13:31:11.805847 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:11.805807    2561 patch_prober.go:28] interesting pod/image-registry-57948c7d77-2m6cf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 13:31:11.805998 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:11.805871    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf" podUID="6a9b257a-51d8-45a0-b821-87d616643748" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:31:21.805689 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:21.805660    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:31:30.662809 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:30.662770    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" podUID="79709d16-5291-4c59-aab6-be8d72936b74" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:31:36.821019 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:36.820965    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf" podUID="6a9b257a-51d8-45a0-b821-87d616643748" containerName="registry" containerID="cri-o://ea27efd53ed54bce9bba4f0a5de5709c875f1f47c8a1e57b0ddfb4b50233b305" gracePeriod=30
Sep 08 13:31:36.962831 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:36.962789    2561 generic.go:356] "Generic (PLEG): container finished" podID="6a9b257a-51d8-45a0-b821-87d616643748" containerID="ea27efd53ed54bce9bba4f0a5de5709c875f1f47c8a1e57b0ddfb4b50233b305" exitCode=0
Sep 08 13:31:36.962990 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:36.962856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf" event={"ID":"6a9b257a-51d8-45a0-b821-87d616643748","Type":"ContainerDied","Data":"ea27efd53ed54bce9bba4f0a5de5709c875f1f47c8a1e57b0ddfb4b50233b305"}
Sep 08 13:31:37.057460 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.057438    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:31:37.140418 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140304    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-trusted-ca\") pod \"6a9b257a-51d8-45a0-b821-87d616643748\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") "
Sep 08 13:31:37.140418 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140347    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-registry-certificates\") pod \"6a9b257a-51d8-45a0-b821-87d616643748\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") "
Sep 08 13:31:37.140418 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140403    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-bound-sa-token\") pod \"6a9b257a-51d8-45a0-b821-87d616643748\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") "
Sep 08 13:31:37.140695 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140502    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") pod \"6a9b257a-51d8-45a0-b821-87d616643748\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") "
Sep 08 13:31:37.140695 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140551    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-image-registry-private-configuration\") pod \"6a9b257a-51d8-45a0-b821-87d616643748\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") "
Sep 08 13:31:37.140695 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140579    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a9b257a-51d8-45a0-b821-87d616643748-ca-trust-extracted\") pod \"6a9b257a-51d8-45a0-b821-87d616643748\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") "
Sep 08 13:31:37.140695 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140609    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-installation-pull-secrets\") pod \"6a9b257a-51d8-45a0-b821-87d616643748\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") "
Sep 08 13:31:37.140695 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdnsm\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-kube-api-access-mdnsm\") pod \"6a9b257a-51d8-45a0-b821-87d616643748\" (UID: \"6a9b257a-51d8-45a0-b821-87d616643748\") "
Sep 08 13:31:37.140948 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6a9b257a-51d8-45a0-b821-87d616643748" (UID: "6a9b257a-51d8-45a0-b821-87d616643748"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:37.140948 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.140913    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6a9b257a-51d8-45a0-b821-87d616643748" (UID: "6a9b257a-51d8-45a0-b821-87d616643748"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:37.142942 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.142917    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-kube-api-access-mdnsm" (OuterVolumeSpecName: "kube-api-access-mdnsm") pod "6a9b257a-51d8-45a0-b821-87d616643748" (UID: "6a9b257a-51d8-45a0-b821-87d616643748"). InnerVolumeSpecName "kube-api-access-mdnsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:37.143056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.142936    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6a9b257a-51d8-45a0-b821-87d616643748" (UID: "6a9b257a-51d8-45a0-b821-87d616643748"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:37.143146 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.143124    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6a9b257a-51d8-45a0-b821-87d616643748" (UID: "6a9b257a-51d8-45a0-b821-87d616643748"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:37.143252 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.143226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6a9b257a-51d8-45a0-b821-87d616643748" (UID: "6a9b257a-51d8-45a0-b821-87d616643748"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:31:37.143311 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.143272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6a9b257a-51d8-45a0-b821-87d616643748" (UID: "6a9b257a-51d8-45a0-b821-87d616643748"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:31:37.149604 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.149575    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9b257a-51d8-45a0-b821-87d616643748-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6a9b257a-51d8-45a0-b821-87d616643748" (UID: "6a9b257a-51d8-45a0-b821-87d616643748"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:37.241489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.241449    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-bound-sa-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:31:37.241489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.241478    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-registry-tls\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:31:37.241489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.241489    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-image-registry-private-configuration\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:31:37.241489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.241499    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a9b257a-51d8-45a0-b821-87d616643748-ca-trust-extracted\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:31:37.241756 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.241509    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a9b257a-51d8-45a0-b821-87d616643748-installation-pull-secrets\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:31:37.241756 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.241517    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdnsm\" (UniqueName: \"kubernetes.io/projected/6a9b257a-51d8-45a0-b821-87d616643748-kube-api-access-mdnsm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:31:37.241756 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.241526    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-trusted-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:31:37.241756 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.241533    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a9b257a-51d8-45a0-b821-87d616643748-registry-certificates\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:31:37.968389 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.968359    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf"
Sep 08 13:31:37.968809 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.968358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57948c7d77-2m6cf" event={"ID":"6a9b257a-51d8-45a0-b821-87d616643748","Type":"ContainerDied","Data":"b39ceaa7900c8ef196134b28813d7b60cfbf63018cbc4c53e0b39707f7beabdc"}
Sep 08 13:31:37.968809 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.968499    2561 scope.go:117] "RemoveContainer" containerID="ea27efd53ed54bce9bba4f0a5de5709c875f1f47c8a1e57b0ddfb4b50233b305"
Sep 08 13:31:37.990168 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.990145    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57948c7d77-2m6cf"]
Sep 08 13:31:37.994328 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:37.994306    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57948c7d77-2m6cf"]
Sep 08 13:31:38.244147 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:38.244065    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a9b257a-51d8-45a0-b821-87d616643748" path="/var/lib/kubelet/pods/6a9b257a-51d8-45a0-b821-87d616643748/volumes"
Sep 08 13:31:40.662842 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:40.662803    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" podUID="79709d16-5291-4c59-aab6-be8d72936b74" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:31:50.662611 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:50.662560    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" podUID="79709d16-5291-4c59-aab6-be8d72936b74" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:31:50.663057 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:50.662649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg"
Sep 08 13:31:50.663212 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:50.663183    2561 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"68fb0109dd1606b4d21b870ed92a72f9dd7c67f144a997351260642c3cd1c183"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 08 13:31:50.663255 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:50.663235    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" podUID="79709d16-5291-4c59-aab6-be8d72936b74" containerName="service-proxy" containerID="cri-o://68fb0109dd1606b4d21b870ed92a72f9dd7c67f144a997351260642c3cd1c183" gracePeriod=30
Sep 08 13:31:51.014213 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:51.014182    2561 generic.go:356] "Generic (PLEG): container finished" podID="79709d16-5291-4c59-aab6-be8d72936b74" containerID="68fb0109dd1606b4d21b870ed92a72f9dd7c67f144a997351260642c3cd1c183" exitCode=2
Sep 08 13:31:51.014403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:51.014250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" event={"ID":"79709d16-5291-4c59-aab6-be8d72936b74","Type":"ContainerDied","Data":"68fb0109dd1606b4d21b870ed92a72f9dd7c67f144a997351260642c3cd1c183"}
Sep 08 13:31:51.014403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:31:51.014298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b4d66b68b-jz6lg" event={"ID":"79709d16-5291-4c59-aab6-be8d72936b74","Type":"ContainerStarted","Data":"0c6bf572b1c6a82abdc0dddea9e953bdb68450093f23070e4fed2a3623280aa7"}
Sep 08 13:32:23.983934 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:23.983892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:32:23.986198 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:23.986178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6404c150-528e-4903-8049-9fa576f8f9ea-metrics-certs\") pod \"network-metrics-daemon-zspg2\" (UID: \"6404c150-528e-4903-8049-9fa576f8f9ea\") " pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:32:24.142356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:24.142326    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-56h6n\""
Sep 08 13:32:24.150541 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:24.150520    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zspg2"
Sep 08 13:32:24.416813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:24.416783    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zspg2"]
Sep 08 13:32:25.126979 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:25.126938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zspg2" event={"ID":"6404c150-528e-4903-8049-9fa576f8f9ea","Type":"ContainerStarted","Data":"a164d44495c3a4fb37237576b634034d873b2a8c04202bdcc5b7aa5d608cd7b6"}
Sep 08 13:32:26.132527 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:26.132492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zspg2" event={"ID":"6404c150-528e-4903-8049-9fa576f8f9ea","Type":"ContainerStarted","Data":"f56f3e9c1eafe17a2151a0b6533d9f1dab496be0ef71f6d6a6bc550d4bb30c5b"}
Sep 08 13:32:26.132918 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:26.132559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zspg2" event={"ID":"6404c150-528e-4903-8049-9fa576f8f9ea","Type":"ContainerStarted","Data":"360e4f9f572932c8c6ea2758c06ea9a739ba6fb4006db4d333dd006b0abbb54c"}
Sep 08 13:32:26.149645 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:32:26.149594    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zspg2" podStartSLOduration=253.317143486 podStartE2EDuration="4m14.149577219s" totalImagesPullingTime="832.433733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:32:26.148512842 +0000 UTC m=+254.499498791" watchObservedRunningTime="2026-09-08 13:32:26.149577219 +0000 UTC m=+254.500563167"
Sep 08 13:33:12.069422 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:33:12.069397    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:35:52.199717 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.199681    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-df6zp"]
Sep 08 13:35:52.200218 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.199906    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9b257a-51d8-45a0-b821-87d616643748" containerName="registry"
Sep 08 13:35:52.200218 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.199916    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9b257a-51d8-45a0-b821-87d616643748" containerName="registry"
Sep 08 13:35:52.200218 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.199964    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9b257a-51d8-45a0-b821-87d616643748" containerName="registry"
Sep 08 13:35:52.202489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.202473    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:52.205912 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.205893    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ftjrn\""
Sep 08 13:35:52.207027 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.207012    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 08 13:35:52.207146 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.207123    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 08 13:35:52.217480 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.217451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-df6zp"]
Sep 08 13:35:52.270396 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.270351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-df6zp\" (UID: \"aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:52.270396 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.270402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdw7p\" (UniqueName: \"kubernetes.io/projected/aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2-kube-api-access-gdw7p\") pod \"cert-manager-webhook-6486ffd997-df6zp\" (UID: \"aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:52.371293 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.371251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-df6zp\" (UID: \"aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:52.371508 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.371319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdw7p\" (UniqueName: \"kubernetes.io/projected/aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2-kube-api-access-gdw7p\") pod \"cert-manager-webhook-6486ffd997-df6zp\" (UID: \"aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:52.379641 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.379611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-df6zp\" (UID: \"aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:52.379767 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.379664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdw7p\" (UniqueName: \"kubernetes.io/projected/aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2-kube-api-access-gdw7p\") pod \"cert-manager-webhook-6486ffd997-df6zp\" (UID: \"aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:52.511491 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.511402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:52.776835 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.776809    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-df6zp"]
Sep 08 13:35:52.779253 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:35:52.779223    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa1d6f37_45af_4cf8_8d56_f8e2b7a9ced2.slice/crio-63b1552aac40789420f01dee7583df305de9961bca81cd7cb30a46519dd63942 WatchSource:0}: Error finding container 63b1552aac40789420f01dee7583df305de9961bca81cd7cb30a46519dd63942: Status 404 returned error can't find the container with id 63b1552aac40789420f01dee7583df305de9961bca81cd7cb30a46519dd63942
Sep 08 13:35:52.781279 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.781264    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:35:52.787825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:52.787800    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp" event={"ID":"aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2","Type":"ContainerStarted","Data":"63b1552aac40789420f01dee7583df305de9961bca81cd7cb30a46519dd63942"}
Sep 08 13:35:56.805127 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:56.805090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp" event={"ID":"aa1d6f37-45af-4cf8-8d56-f8e2b7a9ced2","Type":"ContainerStarted","Data":"a4f157fd10571442c5d712bb204a318445f844735d263038e35085c90fd22e30"}
Sep 08 13:35:56.805596 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:56.805151    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:35:56.822359 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:35:56.822306    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp" podStartSLOduration=1.211635118 podStartE2EDuration="4.82228969s" totalImagesPullingTime="3.610654572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:35:56.821347139 +0000 UTC m=+465.172333089" watchObservedRunningTime="2026-09-08 13:35:56.82228969 +0000 UTC m=+465.173275638"
Sep 08 13:36:02.811549 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:02.811516    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-df6zp"
Sep 08 13:36:06.007929 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.007894    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-c8bsl"]
Sep 08 13:36:06.010693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.010676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-c8bsl"
Sep 08 13:36:06.013108 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.013090    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-g2pkv\""
Sep 08 13:36:06.019343 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.019320    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-c8bsl"]
Sep 08 13:36:06.065351 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.065309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd38d34d-ed67-4d3d-8d2a-679c15d4199e-bound-sa-token\") pod \"cert-manager-55b4d954d5-c8bsl\" (UID: \"bd38d34d-ed67-4d3d-8d2a-679c15d4199e\") " pod="cert-manager/cert-manager-55b4d954d5-c8bsl"
Sep 08 13:36:06.065519 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.065408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8r7\" (UniqueName: \"kubernetes.io/projected/bd38d34d-ed67-4d3d-8d2a-679c15d4199e-kube-api-access-qj8r7\") pod \"cert-manager-55b4d954d5-c8bsl\" (UID: \"bd38d34d-ed67-4d3d-8d2a-679c15d4199e\") " pod="cert-manager/cert-manager-55b4d954d5-c8bsl"
Sep 08 13:36:06.166676 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.166635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8r7\" (UniqueName: \"kubernetes.io/projected/bd38d34d-ed67-4d3d-8d2a-679c15d4199e-kube-api-access-qj8r7\") pod \"cert-manager-55b4d954d5-c8bsl\" (UID: \"bd38d34d-ed67-4d3d-8d2a-679c15d4199e\") " pod="cert-manager/cert-manager-55b4d954d5-c8bsl"
Sep 08 13:36:06.166821 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.166703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd38d34d-ed67-4d3d-8d2a-679c15d4199e-bound-sa-token\") pod \"cert-manager-55b4d954d5-c8bsl\" (UID: \"bd38d34d-ed67-4d3d-8d2a-679c15d4199e\") " pod="cert-manager/cert-manager-55b4d954d5-c8bsl"
Sep 08 13:36:06.175974 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.175951    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd38d34d-ed67-4d3d-8d2a-679c15d4199e-bound-sa-token\") pod \"cert-manager-55b4d954d5-c8bsl\" (UID: \"bd38d34d-ed67-4d3d-8d2a-679c15d4199e\") " pod="cert-manager/cert-manager-55b4d954d5-c8bsl"
Sep 08 13:36:06.176084 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.176024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8r7\" (UniqueName: \"kubernetes.io/projected/bd38d34d-ed67-4d3d-8d2a-679c15d4199e-kube-api-access-qj8r7\") pod \"cert-manager-55b4d954d5-c8bsl\" (UID: \"bd38d34d-ed67-4d3d-8d2a-679c15d4199e\") " pod="cert-manager/cert-manager-55b4d954d5-c8bsl"
Sep 08 13:36:06.320057 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.319968    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-c8bsl"
Sep 08 13:36:06.594874 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.594796    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-c8bsl"]
Sep 08 13:36:06.598210 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:36:06.598186    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd38d34d_ed67_4d3d_8d2a_679c15d4199e.slice/crio-16940407ebe4c88bd50f943e03660bffbac8ec24b582b48fd86ac2c53ca7695e WatchSource:0}: Error finding container 16940407ebe4c88bd50f943e03660bffbac8ec24b582b48fd86ac2c53ca7695e: Status 404 returned error can't find the container with id 16940407ebe4c88bd50f943e03660bffbac8ec24b582b48fd86ac2c53ca7695e
Sep 08 13:36:06.839274 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.839230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-c8bsl" event={"ID":"bd38d34d-ed67-4d3d-8d2a-679c15d4199e","Type":"ContainerStarted","Data":"37f5b1036ee7afbd55bb57d9bab14e0587eefd3be2dad8386557cdf1bdb53697"}
Sep 08 13:36:06.839274 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.839279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-c8bsl" event={"ID":"bd38d34d-ed67-4d3d-8d2a-679c15d4199e","Type":"ContainerStarted","Data":"16940407ebe4c88bd50f943e03660bffbac8ec24b582b48fd86ac2c53ca7695e"}
Sep 08 13:36:06.859274 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:06.859168    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-c8bsl" podStartSLOduration=1.859148054 podStartE2EDuration="1.859148054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:06.858454839 +0000 UTC m=+475.209440787" watchObservedRunningTime="2026-09-08 13:36:06.859148054 +0000 UTC m=+475.210134005"
Sep 08 13:36:21.128667 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.128628    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"]
Sep 08 13:36:21.131725 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.131708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.135652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.135629    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 13:36:21.140285 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.140264    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 13:36:21.141409 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.141388    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 13:36:21.141626 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.141611    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 13:36:21.142535 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.142520    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cdfft\""
Sep 08 13:36:21.174811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.174780    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"]
Sep 08 13:36:21.272792 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.272756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/069ade53-d205-44e5-9eef-7d7e5cef93a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.272792 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.272803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/069ade53-d205-44e5-9eef-7d7e5cef93a8-tmp\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.272999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.272854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/069ade53-d205-44e5-9eef-7d7e5cef93a8-manifests\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.272999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.272883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/069ade53-d205-44e5-9eef-7d7e5cef93a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.272999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.272964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxml\" (UniqueName: \"kubernetes.io/projected/069ade53-d205-44e5-9eef-7d7e5cef93a8-kube-api-access-dfxml\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.374209 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.374170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxml\" (UniqueName: \"kubernetes.io/projected/069ade53-d205-44e5-9eef-7d7e5cef93a8-kube-api-access-dfxml\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.374430 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.374222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/069ade53-d205-44e5-9eef-7d7e5cef93a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.374430 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.374243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/069ade53-d205-44e5-9eef-7d7e5cef93a8-tmp\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.374430 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.374267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/069ade53-d205-44e5-9eef-7d7e5cef93a8-manifests\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.374430 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.374375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/069ade53-d205-44e5-9eef-7d7e5cef93a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.374686 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.374665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/069ade53-d205-44e5-9eef-7d7e5cef93a8-manifests\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.376714 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.376683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/069ade53-d205-44e5-9eef-7d7e5cef93a8-tmp\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.376714 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.376699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/069ade53-d205-44e5-9eef-7d7e5cef93a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.376873 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.376856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/069ade53-d205-44e5-9eef-7d7e5cef93a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.386980 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.386923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxml\" (UniqueName: \"kubernetes.io/projected/069ade53-d205-44e5-9eef-7d7e5cef93a8-kube-api-access-dfxml\") pod \"opendatahub-operator-controller-manager-54685d959b-vzcc8\" (UID: \"069ade53-d205-44e5-9eef-7d7e5cef93a8\") " pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.442011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.441975    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:21.736702 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.736669    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"]
Sep 08 13:36:21.740124 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:36:21.740095    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069ade53_d205_44e5_9eef_7d7e5cef93a8.slice/crio-c0ea00a3e63269438fb4d0e20033c3ae11cab0c9946c140d8fa6c35ddb5889d5 WatchSource:0}: Error finding container c0ea00a3e63269438fb4d0e20033c3ae11cab0c9946c140d8fa6c35ddb5889d5: Status 404 returned error can't find the container with id c0ea00a3e63269438fb4d0e20033c3ae11cab0c9946c140d8fa6c35ddb5889d5
Sep 08 13:36:21.894661 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:21.894621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8" event={"ID":"069ade53-d205-44e5-9eef-7d7e5cef93a8","Type":"ContainerStarted","Data":"c0ea00a3e63269438fb4d0e20033c3ae11cab0c9946c140d8fa6c35ddb5889d5"}
Sep 08 13:36:26.913122 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:26.913081    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8" event={"ID":"069ade53-d205-44e5-9eef-7d7e5cef93a8","Type":"ContainerStarted","Data":"fada984d8a1bf2f1b008713fbc597e704312e44bf2bc7553078386f8c794c26d"}
Sep 08 13:36:27.917562 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:27.917525    2561 generic.go:356] "Generic (PLEG): container finished" podID="069ade53-d205-44e5-9eef-7d7e5cef93a8" containerID="fada984d8a1bf2f1b008713fbc597e704312e44bf2bc7553078386f8c794c26d" exitCode=0
Sep 08 13:36:27.917562 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:27.917568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8" event={"ID":"069ade53-d205-44e5-9eef-7d7e5cef93a8","Type":"ContainerDied","Data":"fada984d8a1bf2f1b008713fbc597e704312e44bf2bc7553078386f8c794c26d"}
Sep 08 13:36:28.927191 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:28.927158    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8" event={"ID":"069ade53-d205-44e5-9eef-7d7e5cef93a8","Type":"ContainerStarted","Data":"3fc654556bcf9f70f785eadb9aeaa5351a508b7d4f12cb5909f8caddace019d7"}
Sep 08 13:36:28.927587 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:28.927251    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:39.932443 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:39.932409    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8"
Sep 08 13:36:39.960806 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:39.960750    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54685d959b-vzcc8" podStartSLOduration=13.259568335 podStartE2EDuration="18.96073014s" totalImagesPullingTime="4.611314724s" totalInitContainerRuntime="1.089847081s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:36:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:29.007512682 +0000 UTC m=+497.358498629" watchObservedRunningTime="2026-09-08 13:36:39.96073014 +0000 UTC m=+508.311716173"
Sep 08 13:36:48.159577 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.159539    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55db547db7-tslwp"]
Sep 08 13:36:48.162722 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.162705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.167491 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.167468    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 13:36:48.167861 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.167843    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 13:36:48.167933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.167847    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 13:36:48.168409 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.168366    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 13:36:48.169737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.169722    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lxrw9\""
Sep 08 13:36:48.181243 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.181220    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55db547db7-tslwp"]
Sep 08 13:36:48.274156 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.274115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b092c6-3ec8-4550-9aea-5354fe082416-tls-certs\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.274352 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.274173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdld5\" (UniqueName: \"kubernetes.io/projected/a6b092c6-3ec8-4550-9aea-5354fe082416-kube-api-access-wdld5\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.274352 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.274244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6b092c6-3ec8-4550-9aea-5354fe082416-tmp\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.375039 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.374996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6b092c6-3ec8-4550-9aea-5354fe082416-tmp\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.375201 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.375059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b092c6-3ec8-4550-9aea-5354fe082416-tls-certs\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.375201 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.375092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdld5\" (UniqueName: \"kubernetes.io/projected/a6b092c6-3ec8-4550-9aea-5354fe082416-kube-api-access-wdld5\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.375201 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:36:48.375191    2561 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 08 13:36:48.375303 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:36:48.375270    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6b092c6-3ec8-4550-9aea-5354fe082416-tls-certs podName:a6b092c6-3ec8-4550-9aea-5354fe082416 nodeName:}" failed. No retries permitted until 2026-09-08 13:36:48.875253426 +0000 UTC m=+517.226239352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a6b092c6-3ec8-4550-9aea-5354fe082416-tls-certs") pod "kube-auth-proxy-55db547db7-tslwp" (UID: "a6b092c6-3ec8-4550-9aea-5354fe082416") : secret "kube-auth-proxy-tls" not found
Sep 08 13:36:48.377205 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.377178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6b092c6-3ec8-4550-9aea-5354fe082416-tmp\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.391197 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.391170    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdld5\" (UniqueName: \"kubernetes.io/projected/a6b092c6-3ec8-4550-9aea-5354fe082416-kube-api-access-wdld5\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.879922 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.879875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b092c6-3ec8-4550-9aea-5354fe082416-tls-certs\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:48.882187 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:48.882153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b092c6-3ec8-4550-9aea-5354fe082416-tls-certs\") pod \"kube-auth-proxy-55db547db7-tslwp\" (UID: \"a6b092c6-3ec8-4550-9aea-5354fe082416\") " pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:49.071878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:49.071843    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp"
Sep 08 13:36:49.373928 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:49.373893    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55db547db7-tslwp"]
Sep 08 13:36:49.376978 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:36:49.376951    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b092c6_3ec8_4550_9aea_5354fe082416.slice/crio-f9cb88636c7f70590c763d49439ad3b7607c738732d0841b598a8545177cc12e WatchSource:0}: Error finding container f9cb88636c7f70590c763d49439ad3b7607c738732d0841b598a8545177cc12e: Status 404 returned error can't find the container with id f9cb88636c7f70590c763d49439ad3b7607c738732d0841b598a8545177cc12e
Sep 08 13:36:49.999054 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:49.999023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp" event={"ID":"a6b092c6-3ec8-4550-9aea-5354fe082416","Type":"ContainerStarted","Data":"f9cb88636c7f70590c763d49439ad3b7607c738732d0841b598a8545177cc12e"}
Sep 08 13:36:53.015221 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.015129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp" event={"ID":"a6b092c6-3ec8-4550-9aea-5354fe082416","Type":"ContainerStarted","Data":"79eb483e32faaa97a9d5b03e1a4bf0d4f16188a26b23066dba31c199bf473f21"}
Sep 08 13:36:53.079951 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.077130    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55db547db7-tslwp" podStartSLOduration=1.764823773 podStartE2EDuration="5.077101584s" totalImagesPullingTime="3.312277811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:53.075282688 +0000 UTC m=+521.426268636" watchObservedRunningTime="2026-09-08 13:36:53.077101584 +0000 UTC m=+521.428087532"
Sep 08 13:36:53.555486 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.555456    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-75wrn"]
Sep 08 13:36:53.558532 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.558516    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:53.561372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.561353    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 08 13:36:53.564469 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.564290    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gpjjq\""
Sep 08 13:36:53.586802 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.586776    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-75wrn"]
Sep 08 13:36:53.613884 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.613858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f78383c-61b1-4516-afce-f6dd94051218-cert\") pod \"odh-model-controller-7998c67dc4-75wrn\" (UID: \"0f78383c-61b1-4516-afce-f6dd94051218\") " pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:53.614002 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.613894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4xtv\" (UniqueName: \"kubernetes.io/projected/0f78383c-61b1-4516-afce-f6dd94051218-kube-api-access-m4xtv\") pod \"odh-model-controller-7998c67dc4-75wrn\" (UID: \"0f78383c-61b1-4516-afce-f6dd94051218\") " pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:53.714355 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.714313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f78383c-61b1-4516-afce-f6dd94051218-cert\") pod \"odh-model-controller-7998c67dc4-75wrn\" (UID: \"0f78383c-61b1-4516-afce-f6dd94051218\") " pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:53.714559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.714398    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4xtv\" (UniqueName: \"kubernetes.io/projected/0f78383c-61b1-4516-afce-f6dd94051218-kube-api-access-m4xtv\") pod \"odh-model-controller-7998c67dc4-75wrn\" (UID: \"0f78383c-61b1-4516-afce-f6dd94051218\") " pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:53.714559 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:36:53.714484    2561 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 08 13:36:53.714675 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:36:53.714561    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f78383c-61b1-4516-afce-f6dd94051218-cert podName:0f78383c-61b1-4516-afce-f6dd94051218 nodeName:}" failed. No retries permitted until 2026-09-08 13:36:54.214538598 +0000 UTC m=+522.565524544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0f78383c-61b1-4516-afce-f6dd94051218-cert") pod "odh-model-controller-7998c67dc4-75wrn" (UID: "0f78383c-61b1-4516-afce-f6dd94051218") : secret "odh-model-controller-webhook-cert" not found
Sep 08 13:36:53.747726 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:53.747694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4xtv\" (UniqueName: \"kubernetes.io/projected/0f78383c-61b1-4516-afce-f6dd94051218-kube-api-access-m4xtv\") pod \"odh-model-controller-7998c67dc4-75wrn\" (UID: \"0f78383c-61b1-4516-afce-f6dd94051218\") " pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:54.217394 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.217348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f78383c-61b1-4516-afce-f6dd94051218-cert\") pod \"odh-model-controller-7998c67dc4-75wrn\" (UID: \"0f78383c-61b1-4516-afce-f6dd94051218\") " pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:54.219672 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.219654    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f78383c-61b1-4516-afce-f6dd94051218-cert\") pod \"odh-model-controller-7998c67dc4-75wrn\" (UID: \"0f78383c-61b1-4516-afce-f6dd94051218\") " pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:54.467712 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.467633    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:54.517975 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.517948    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p58pj"]
Sep 08 13:36:54.522024 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.521998    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:54.525022 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.524999    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 08 13:36:54.580719 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.580682    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p58pj"]
Sep 08 13:36:54.620431 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.620400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjb4\" (UniqueName: \"kubernetes.io/projected/a1adf55b-2b9e-4a5b-a060-09475b42475c-kube-api-access-7kjb4\") pod \"model-serving-api-54d89bd5fc-p58pj\" (UID: \"a1adf55b-2b9e-4a5b-a060-09475b42475c\") " pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:54.620591 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.620444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1adf55b-2b9e-4a5b-a060-09475b42475c-tls-certs\") pod \"model-serving-api-54d89bd5fc-p58pj\" (UID: \"a1adf55b-2b9e-4a5b-a060-09475b42475c\") " pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:54.720987 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.720915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjb4\" (UniqueName: \"kubernetes.io/projected/a1adf55b-2b9e-4a5b-a060-09475b42475c-kube-api-access-7kjb4\") pod \"model-serving-api-54d89bd5fc-p58pj\" (UID: \"a1adf55b-2b9e-4a5b-a060-09475b42475c\") " pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:54.720987 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.720959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1adf55b-2b9e-4a5b-a060-09475b42475c-tls-certs\") pod \"model-serving-api-54d89bd5fc-p58pj\" (UID: \"a1adf55b-2b9e-4a5b-a060-09475b42475c\") " pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:54.721148 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:36:54.721049    2561 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 08 13:36:54.721148 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:36:54.721104    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1adf55b-2b9e-4a5b-a060-09475b42475c-tls-certs podName:a1adf55b-2b9e-4a5b-a060-09475b42475c nodeName:}" failed. No retries permitted until 2026-09-08 13:36:55.221089544 +0000 UTC m=+523.572075471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a1adf55b-2b9e-4a5b-a060-09475b42475c-tls-certs") pod "model-serving-api-54d89bd5fc-p58pj" (UID: "a1adf55b-2b9e-4a5b-a060-09475b42475c") : secret "model-serving-api-tls" not found
Sep 08 13:36:54.734728 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.734705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjb4\" (UniqueName: \"kubernetes.io/projected/a1adf55b-2b9e-4a5b-a060-09475b42475c-kube-api-access-7kjb4\") pod \"model-serving-api-54d89bd5fc-p58pj\" (UID: \"a1adf55b-2b9e-4a5b-a060-09475b42475c\") " pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:54.746371 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:54.746347    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-75wrn"]
Sep 08 13:36:54.748432 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:36:54.748405    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f78383c_61b1_4516_afce_f6dd94051218.slice/crio-f90fc180b43bf62bf2d746ae1aeb2d2f76596d96b3e9b1b1a00c62050ae51dd5 WatchSource:0}: Error finding container f90fc180b43bf62bf2d746ae1aeb2d2f76596d96b3e9b1b1a00c62050ae51dd5: Status 404 returned error can't find the container with id f90fc180b43bf62bf2d746ae1aeb2d2f76596d96b3e9b1b1a00c62050ae51dd5
Sep 08 13:36:55.023230 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:55.023131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-75wrn" event={"ID":"0f78383c-61b1-4516-afce-f6dd94051218","Type":"ContainerStarted","Data":"f90fc180b43bf62bf2d746ae1aeb2d2f76596d96b3e9b1b1a00c62050ae51dd5"}
Sep 08 13:36:55.224670 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:55.224625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1adf55b-2b9e-4a5b-a060-09475b42475c-tls-certs\") pod \"model-serving-api-54d89bd5fc-p58pj\" (UID: \"a1adf55b-2b9e-4a5b-a060-09475b42475c\") " pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:55.227518 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:55.227491    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1adf55b-2b9e-4a5b-a060-09475b42475c-tls-certs\") pod \"model-serving-api-54d89bd5fc-p58pj\" (UID: \"a1adf55b-2b9e-4a5b-a060-09475b42475c\") " pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:55.435870 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:55.435828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:36:55.733499 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:55.733352    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p58pj"]
Sep 08 13:36:55.739413 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:36:55.739364    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1adf55b_2b9e_4a5b_a060_09475b42475c.slice/crio-3eb4ed84e7b6b3a2c1d00c2338dcd549cf17af98122da8b36e836a82e05a05ae WatchSource:0}: Error finding container 3eb4ed84e7b6b3a2c1d00c2338dcd549cf17af98122da8b36e836a82e05a05ae: Status 404 returned error can't find the container with id 3eb4ed84e7b6b3a2c1d00c2338dcd549cf17af98122da8b36e836a82e05a05ae
Sep 08 13:36:56.029604 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:56.029516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p58pj" event={"ID":"a1adf55b-2b9e-4a5b-a060-09475b42475c","Type":"ContainerStarted","Data":"3eb4ed84e7b6b3a2c1d00c2338dcd549cf17af98122da8b36e836a82e05a05ae"}
Sep 08 13:36:58.041169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:58.041133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-75wrn" event={"ID":"0f78383c-61b1-4516-afce-f6dd94051218","Type":"ContainerStarted","Data":"b29a0e6b2f9ee1dc3216124a7cf914f8823358774d48b28acb30f5b2ec7465bd"}
Sep 08 13:36:58.041617 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:58.041240    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:36:58.074465 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:58.074405    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-75wrn" podStartSLOduration=2.606620356 podStartE2EDuration="5.074366957s" totalImagesPullingTime="2.467746601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:58.072388809 +0000 UTC m=+526.423374754" watchObservedRunningTime="2026-09-08 13:36:58.074366957 +0000 UTC m=+526.425352905"
Sep 08 13:36:59.046518 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.046481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p58pj" event={"ID":"a1adf55b-2b9e-4a5b-a060-09475b42475c","Type":"ContainerStarted","Data":"af007717ef056b01d725b5db291a36912aef1ce319c7e6a36e4d98d354cf40b8"}
Sep 08 13:36:59.070797 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.070752    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-p58pj" podStartSLOduration=2.591193895 podStartE2EDuration="5.070735269s" totalImagesPullingTime="2.479541374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:59.070152774 +0000 UTC m=+527.421138721" watchObservedRunningTime="2026-09-08 13:36:59.070735269 +0000 UTC m=+527.421721216"
Sep 08 13:36:59.371346 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.371272    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"]
Sep 08 13:36:59.374540 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.374524    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.380971 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.380947    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 13:36:59.381075 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.380997    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 13:36:59.382101 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.382083    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:36:59.382183 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.382083    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 13:36:59.382183 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.382089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 13:36:59.382272 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.382186    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rxx8j\""
Sep 08 13:36:59.397738 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.397714    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"]
Sep 08 13:36:59.457078 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.457044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a63233e2-b5f0-471c-b57f-bc5b91441c5f-metrics-cert\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.457078 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.457077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a63233e2-b5f0-471c-b57f-bc5b91441c5f-cert\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.457279 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.457115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a63233e2-b5f0-471c-b57f-bc5b91441c5f-manager-config\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.457279 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.457154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86pm7\" (UniqueName: \"kubernetes.io/projected/a63233e2-b5f0-471c-b57f-bc5b91441c5f-kube-api-access-86pm7\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.558331 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.558289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a63233e2-b5f0-471c-b57f-bc5b91441c5f-manager-config\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.558544 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.558347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86pm7\" (UniqueName: \"kubernetes.io/projected/a63233e2-b5f0-471c-b57f-bc5b91441c5f-kube-api-access-86pm7\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.558544 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.558426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a63233e2-b5f0-471c-b57f-bc5b91441c5f-metrics-cert\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.558544 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.558453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a63233e2-b5f0-471c-b57f-bc5b91441c5f-cert\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.558994 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.558960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a63233e2-b5f0-471c-b57f-bc5b91441c5f-manager-config\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.560931 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.560905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a63233e2-b5f0-471c-b57f-bc5b91441c5f-cert\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.561058 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.560962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a63233e2-b5f0-471c-b57f-bc5b91441c5f-metrics-cert\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.567962 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.567939    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86pm7\" (UniqueName: \"kubernetes.io/projected/a63233e2-b5f0-471c-b57f-bc5b91441c5f-kube-api-access-86pm7\") pod \"lws-controller-manager-8d99975cf-tgbgj\" (UID: \"a63233e2-b5f0-471c-b57f-bc5b91441c5f\") " pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.682681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.682656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:36:59.985005 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:36:59.984982    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"]
Sep 08 13:36:59.986955 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:36:59.986924    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63233e2_b5f0_471c_b57f_bc5b91441c5f.slice/crio-97e5bc5af3553eddd3ac5aade442f3469daf7cb7843782fe961587e62b19433f WatchSource:0}: Error finding container 97e5bc5af3553eddd3ac5aade442f3469daf7cb7843782fe961587e62b19433f: Status 404 returned error can't find the container with id 97e5bc5af3553eddd3ac5aade442f3469daf7cb7843782fe961587e62b19433f
Sep 08 13:37:00.052421 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:37:00.052386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj" event={"ID":"a63233e2-b5f0-471c-b57f-bc5b91441c5f","Type":"ContainerStarted","Data":"97e5bc5af3553eddd3ac5aade442f3469daf7cb7843782fe961587e62b19433f"}
Sep 08 13:37:00.052825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:37:00.052489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:37:03.072807 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:37:03.072770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj" event={"ID":"a63233e2-b5f0-471c-b57f-bc5b91441c5f","Type":"ContainerStarted","Data":"2d50d86679ac2147407fa572f5a813338c7d5061514d341cc8b595e68d10dfe6"}
Sep 08 13:37:03.073245 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:37:03.072852    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:37:03.179268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:37:03.179215    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj" podStartSLOduration=1.938613495 podStartE2EDuration="4.179200252s" totalImagesPullingTime="2.240586757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:37:03.178412647 +0000 UTC m=+531.529398586" watchObservedRunningTime="2026-09-08 13:37:03.179200252 +0000 UTC m=+531.530186200"
Sep 08 13:37:09.048688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:37:09.048616    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-75wrn"
Sep 08 13:37:11.061259 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:37:11.061224    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-p58pj"
Sep 08 13:37:14.079075 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:37:14.079042    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8d99975cf-tgbgj"
Sep 08 13:39:18.933197 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:18.933160    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-g2ttb"]
Sep 08 13:39:18.936566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:18.936550    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-g2ttb"
Sep 08 13:39:18.939443 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:18.939418    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-f8sxc\""
Sep 08 13:39:18.940577 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:18.940556    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 13:39:18.940694 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:18.940616    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 13:39:18.948297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:18.948275    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-g2ttb"]
Sep 08 13:39:19.093027 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.092980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfqz\" (UniqueName: \"kubernetes.io/projected/7e864c3c-ec2d-4a59-8787-d4521a56ab80-kube-api-access-wmfqz\") pod \"authorino-b54df94b4-g2ttb\" (UID: \"7e864c3c-ec2d-4a59-8787-d4521a56ab80\") " pod="kuadrant-system/authorino-b54df94b4-g2ttb"
Sep 08 13:39:19.194414 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.194287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfqz\" (UniqueName: \"kubernetes.io/projected/7e864c3c-ec2d-4a59-8787-d4521a56ab80-kube-api-access-wmfqz\") pod \"authorino-b54df94b4-g2ttb\" (UID: \"7e864c3c-ec2d-4a59-8787-d4521a56ab80\") " pod="kuadrant-system/authorino-b54df94b4-g2ttb"
Sep 08 13:39:19.204389 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.204354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfqz\" (UniqueName: \"kubernetes.io/projected/7e864c3c-ec2d-4a59-8787-d4521a56ab80-kube-api-access-wmfqz\") pod \"authorino-b54df94b4-g2ttb\" (UID: \"7e864c3c-ec2d-4a59-8787-d4521a56ab80\") " pod="kuadrant-system/authorino-b54df94b4-g2ttb"
Sep 08 13:39:19.245013 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.244985    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-g2ttb"
Sep 08 13:39:19.298884 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.298855    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gtqft"]
Sep 08 13:39:19.304402 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.304101    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-gtqft"
Sep 08 13:39:19.310882 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.310853    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gtqft"]
Sep 08 13:39:19.395228 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.395187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqdn\" (UniqueName: \"kubernetes.io/projected/f3a653ff-3c6c-47b7-b09c-ed8b2790767e-kube-api-access-fvqdn\") pod \"authorino-c548b5d7d-gtqft\" (UID: \"f3a653ff-3c6c-47b7-b09c-ed8b2790767e\") " pod="kuadrant-system/authorino-c548b5d7d-gtqft"
Sep 08 13:39:19.496353 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.496274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqdn\" (UniqueName: \"kubernetes.io/projected/f3a653ff-3c6c-47b7-b09c-ed8b2790767e-kube-api-access-fvqdn\") pod \"authorino-c548b5d7d-gtqft\" (UID: \"f3a653ff-3c6c-47b7-b09c-ed8b2790767e\") " pod="kuadrant-system/authorino-c548b5d7d-gtqft"
Sep 08 13:39:19.506993 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.506967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqdn\" (UniqueName: \"kubernetes.io/projected/f3a653ff-3c6c-47b7-b09c-ed8b2790767e-kube-api-access-fvqdn\") pod \"authorino-c548b5d7d-gtqft\" (UID: \"f3a653ff-3c6c-47b7-b09c-ed8b2790767e\") " pod="kuadrant-system/authorino-c548b5d7d-gtqft"
Sep 08 13:39:19.515969 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.515937    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-g2ttb"]
Sep 08 13:39:19.519696 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:39:19.519667    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e864c3c_ec2d_4a59_8787_d4521a56ab80.slice/crio-6be3e8123b7ee5128889161faf0e68d8a2e5e1c69af42e268f9814888eab7bbd WatchSource:0}: Error finding container 6be3e8123b7ee5128889161faf0e68d8a2e5e1c69af42e268f9814888eab7bbd: Status 404 returned error can't find the container with id 6be3e8123b7ee5128889161faf0e68d8a2e5e1c69af42e268f9814888eab7bbd
Sep 08 13:39:19.562610 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.562575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-g2ttb" event={"ID":"7e864c3c-ec2d-4a59-8787-d4521a56ab80","Type":"ContainerStarted","Data":"6be3e8123b7ee5128889161faf0e68d8a2e5e1c69af42e268f9814888eab7bbd"}
Sep 08 13:39:19.616057 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.616017    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-gtqft"
Sep 08 13:39:19.894514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:19.894488    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gtqft"]
Sep 08 13:39:19.896155 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:39:19.896119    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3a653ff_3c6c_47b7_b09c_ed8b2790767e.slice/crio-8972a696a434e3820487738868b7beefd24e5cf14ac34b444a9d7559dacadb06 WatchSource:0}: Error finding container 8972a696a434e3820487738868b7beefd24e5cf14ac34b444a9d7559dacadb06: Status 404 returned error can't find the container with id 8972a696a434e3820487738868b7beefd24e5cf14ac34b444a9d7559dacadb06
Sep 08 13:39:20.567929 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:20.567895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-gtqft" event={"ID":"f3a653ff-3c6c-47b7-b09c-ed8b2790767e","Type":"ContainerStarted","Data":"8972a696a434e3820487738868b7beefd24e5cf14ac34b444a9d7559dacadb06"}
Sep 08 13:39:23.583949 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:23.583912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-g2ttb" event={"ID":"7e864c3c-ec2d-4a59-8787-d4521a56ab80","Type":"ContainerStarted","Data":"777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562"}
Sep 08 13:39:23.600854 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:23.600803    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-g2ttb" podStartSLOduration=2.55987324 podStartE2EDuration="5.600788006s" totalImagesPullingTime="3.040914766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:39:23.599345069 +0000 UTC m=+671.950331017" watchObservedRunningTime="2026-09-08 13:39:23.600788006 +0000 UTC m=+671.951773954"
Sep 08 13:39:29.608290 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:29.608252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-gtqft" event={"ID":"f3a653ff-3c6c-47b7-b09c-ed8b2790767e","Type":"ContainerStarted","Data":"26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4"}
Sep 08 13:39:29.625288 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:29.625237    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-gtqft" podStartSLOduration=1.727360278 podStartE2EDuration="10.625221541s" totalImagesPullingTime="8.897861263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:39:29.624354685 +0000 UTC m=+677.975340636" watchObservedRunningTime="2026-09-08 13:39:29.625221541 +0000 UTC m=+677.976207550"
Sep 08 13:39:29.652876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:29.652840    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-g2ttb"]
Sep 08 13:39:29.653081 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:29.653052    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-g2ttb" podUID="7e864c3c-ec2d-4a59-8787-d4521a56ab80" containerName="authorino" containerID="cri-o://777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562" gracePeriod=30
Sep 08 13:39:29.890545 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:29.890519    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-g2ttb"
Sep 08 13:39:30.077878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.077841    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmfqz\" (UniqueName: \"kubernetes.io/projected/7e864c3c-ec2d-4a59-8787-d4521a56ab80-kube-api-access-wmfqz\") pod \"7e864c3c-ec2d-4a59-8787-d4521a56ab80\" (UID: \"7e864c3c-ec2d-4a59-8787-d4521a56ab80\") "
Sep 08 13:39:30.079993 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.079959    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e864c3c-ec2d-4a59-8787-d4521a56ab80-kube-api-access-wmfqz" (OuterVolumeSpecName: "kube-api-access-wmfqz") pod "7e864c3c-ec2d-4a59-8787-d4521a56ab80" (UID: "7e864c3c-ec2d-4a59-8787-d4521a56ab80"). InnerVolumeSpecName "kube-api-access-wmfqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:30.178613 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.178583    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmfqz\" (UniqueName: \"kubernetes.io/projected/7e864c3c-ec2d-4a59-8787-d4521a56ab80-kube-api-access-wmfqz\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:39:30.613328 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.613235    2561 generic.go:356] "Generic (PLEG): container finished" podID="7e864c3c-ec2d-4a59-8787-d4521a56ab80" containerID="777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562" exitCode=0
Sep 08 13:39:30.613328 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.613280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-g2ttb"
Sep 08 13:39:30.613845 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.613324    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-g2ttb" event={"ID":"7e864c3c-ec2d-4a59-8787-d4521a56ab80","Type":"ContainerDied","Data":"777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562"}
Sep 08 13:39:30.613845 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.613366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-g2ttb" event={"ID":"7e864c3c-ec2d-4a59-8787-d4521a56ab80","Type":"ContainerDied","Data":"6be3e8123b7ee5128889161faf0e68d8a2e5e1c69af42e268f9814888eab7bbd"}
Sep 08 13:39:30.613845 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.613397    2561 scope.go:117] "RemoveContainer" containerID="777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562"
Sep 08 13:39:30.621748 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.621733    2561 scope.go:117] "RemoveContainer" containerID="777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562"
Sep 08 13:39:30.622052 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:39:30.622029    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562\": container with ID starting with 777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562 not found: ID does not exist" containerID="777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562"
Sep 08 13:39:30.622148 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.622056    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562"} err="failed to get container status \"777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562\": rpc error: code = NotFound desc = could not find container \"777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562\": container with ID starting with 777b40bceeca618d9b32d53ce1edc7831cd4856dfd483b37a6e23c3b5386d562 not found: ID does not exist"
Sep 08 13:39:30.630982 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.630960    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-g2ttb"]
Sep 08 13:39:30.634594 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:30.634573    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-g2ttb"]
Sep 08 13:39:32.243888 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:32.243855    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e864c3c-ec2d-4a59-8787-d4521a56ab80" path="/var/lib/kubelet/pods/7e864c3c-ec2d-4a59-8787-d4521a56ab80/volumes"
Sep 08 13:39:49.596204 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.596170    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-vszdj"]
Sep 08 13:39:49.596624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.596447    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e864c3c-ec2d-4a59-8787-d4521a56ab80" containerName="authorino"
Sep 08 13:39:49.596624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.596459    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e864c3c-ec2d-4a59-8787-d4521a56ab80" containerName="authorino"
Sep 08 13:39:49.596624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.596505    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e864c3c-ec2d-4a59-8787-d4521a56ab80" containerName="authorino"
Sep 08 13:39:49.599164 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.599150    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj"
Sep 08 13:39:49.612318 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.612291    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-vszdj"]
Sep 08 13:39:49.620500 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.620475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qph\" (UniqueName: \"kubernetes.io/projected/f1a9e050-e130-4f2d-aad9-58c8d13bf17d-kube-api-access-b2qph\") pod \"authorino-5dc7ccb5f-vszdj\" (UID: \"f1a9e050-e130-4f2d-aad9-58c8d13bf17d\") " pod="kuadrant-system/authorino-5dc7ccb5f-vszdj"
Sep 08 13:39:49.720989 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.720955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qph\" (UniqueName: \"kubernetes.io/projected/f1a9e050-e130-4f2d-aad9-58c8d13bf17d-kube-api-access-b2qph\") pod \"authorino-5dc7ccb5f-vszdj\" (UID: \"f1a9e050-e130-4f2d-aad9-58c8d13bf17d\") " pod="kuadrant-system/authorino-5dc7ccb5f-vszdj"
Sep 08 13:39:49.731538 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.731505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qph\" (UniqueName: \"kubernetes.io/projected/f1a9e050-e130-4f2d-aad9-58c8d13bf17d-kube-api-access-b2qph\") pod \"authorino-5dc7ccb5f-vszdj\" (UID: \"f1a9e050-e130-4f2d-aad9-58c8d13bf17d\") " pod="kuadrant-system/authorino-5dc7ccb5f-vszdj"
Sep 08 13:39:49.855426 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.855325    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-vszdj"]
Sep 08 13:39:49.855571 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.855551    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj"
Sep 08 13:39:49.895121 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.895084    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69554c6566-xdwcg"]
Sep 08 13:39:49.898501 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.898485    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69554c6566-xdwcg"
Sep 08 13:39:49.913477 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.913115    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69554c6566-xdwcg"]
Sep 08 13:39:49.922689 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:49.922644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7qn\" (UniqueName: \"kubernetes.io/projected/2c2dfa3e-cbda-42e2-81c3-66f7a16887d5-kube-api-access-7l7qn\") pod \"authorino-69554c6566-xdwcg\" (UID: \"2c2dfa3e-cbda-42e2-81c3-66f7a16887d5\") " pod="kuadrant-system/authorino-69554c6566-xdwcg"
Sep 08 13:39:50.015157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.015125    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69554c6566-xdwcg"]
Sep 08 13:39:50.015374 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:39:50.015352    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7l7qn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-69554c6566-xdwcg" podUID="2c2dfa3e-cbda-42e2-81c3-66f7a16887d5"
Sep 08 13:39:50.023257 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.023234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7qn\" (UniqueName: \"kubernetes.io/projected/2c2dfa3e-cbda-42e2-81c3-66f7a16887d5-kube-api-access-7l7qn\") pod \"authorino-69554c6566-xdwcg\" (UID: \"2c2dfa3e-cbda-42e2-81c3-66f7a16887d5\") " pod="kuadrant-system/authorino-69554c6566-xdwcg"
Sep 08 13:39:50.040509 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.040476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7qn\" (UniqueName: \"kubernetes.io/projected/2c2dfa3e-cbda-42e2-81c3-66f7a16887d5-kube-api-access-7l7qn\") pod \"authorino-69554c6566-xdwcg\" (UID: \"2c2dfa3e-cbda-42e2-81c3-66f7a16887d5\") " pod="kuadrant-system/authorino-69554c6566-xdwcg"
Sep 08 13:39:50.044402 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.044359    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f64b7bcfb-2s7lb"]
Sep 08 13:39:50.047472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.047454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:39:50.050145 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.050126    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 13:39:50.056323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.056299    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f64b7bcfb-2s7lb"]
Sep 08 13:39:50.124032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.123951    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4p8b\" (UniqueName: \"kubernetes.io/projected/88c3f4d7-a5cc-44f7-9f4d-605770334df0-kube-api-access-p4p8b\") pod \"authorino-5f64b7bcfb-2s7lb\" (UID: \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\") " pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:39:50.124032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.123986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88c3f4d7-a5cc-44f7-9f4d-605770334df0-tls-cert\") pod \"authorino-5f64b7bcfb-2s7lb\" (UID: \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\") " pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:39:50.127310 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.127284    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-vszdj"]
Sep 08 13:39:50.130335 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:39:50.130306    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a9e050_e130_4f2d_aad9_58c8d13bf17d.slice/crio-e7a5925f66878767302d65c3549fc375a9ddc9738d36a0c89839eccfe4e365e6 WatchSource:0}: Error finding container e7a5925f66878767302d65c3549fc375a9ddc9738d36a0c89839eccfe4e365e6: Status 404 returned error can't find the container with id e7a5925f66878767302d65c3549fc375a9ddc9738d36a0c89839eccfe4e365e6
Sep 08 13:39:50.224706 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.224668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4p8b\" (UniqueName: \"kubernetes.io/projected/88c3f4d7-a5cc-44f7-9f4d-605770334df0-kube-api-access-p4p8b\") pod \"authorino-5f64b7bcfb-2s7lb\" (UID: \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\") " pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:39:50.224706 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.224707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88c3f4d7-a5cc-44f7-9f4d-605770334df0-tls-cert\") pod \"authorino-5f64b7bcfb-2s7lb\" (UID: \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\") " pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:39:50.227177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.227155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88c3f4d7-a5cc-44f7-9f4d-605770334df0-tls-cert\") pod \"authorino-5f64b7bcfb-2s7lb\" (UID: \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\") " pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:39:50.234746 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.234728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4p8b\" (UniqueName: \"kubernetes.io/projected/88c3f4d7-a5cc-44f7-9f4d-605770334df0-kube-api-access-p4p8b\") pod \"authorino-5f64b7bcfb-2s7lb\" (UID: \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\") " pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:39:50.356718 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.356680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:39:50.627747 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.627666    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f64b7bcfb-2s7lb"]
Sep 08 13:39:50.630227 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:39:50.630200    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c3f4d7_a5cc_44f7_9f4d_605770334df0.slice/crio-392d2fd14f92898497df7acf9021050bbf832d885a0c5843e0b0d05f123e4ee4 WatchSource:0}: Error finding container 392d2fd14f92898497df7acf9021050bbf832d885a0c5843e0b0d05f123e4ee4: Status 404 returned error can't find the container with id 392d2fd14f92898497df7acf9021050bbf832d885a0c5843e0b0d05f123e4ee4
Sep 08 13:39:50.690997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.690958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb" event={"ID":"88c3f4d7-a5cc-44f7-9f4d-605770334df0","Type":"ContainerStarted","Data":"392d2fd14f92898497df7acf9021050bbf832d885a0c5843e0b0d05f123e4ee4"}
Sep 08 13:39:50.692117 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.692092    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69554c6566-xdwcg"
Sep 08 13:39:50.692285 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.692113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj" event={"ID":"f1a9e050-e130-4f2d-aad9-58c8d13bf17d","Type":"ContainerStarted","Data":"e7a5925f66878767302d65c3549fc375a9ddc9738d36a0c89839eccfe4e365e6"}
Sep 08 13:39:50.699799 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.699780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69554c6566-xdwcg"
Sep 08 13:39:50.729214 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.729185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l7qn\" (UniqueName: \"kubernetes.io/projected/2c2dfa3e-cbda-42e2-81c3-66f7a16887d5-kube-api-access-7l7qn\") pod \"2c2dfa3e-cbda-42e2-81c3-66f7a16887d5\" (UID: \"2c2dfa3e-cbda-42e2-81c3-66f7a16887d5\") "
Sep 08 13:39:50.731278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.731255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2dfa3e-cbda-42e2-81c3-66f7a16887d5-kube-api-access-7l7qn" (OuterVolumeSpecName: "kube-api-access-7l7qn") pod "2c2dfa3e-cbda-42e2-81c3-66f7a16887d5" (UID: "2c2dfa3e-cbda-42e2-81c3-66f7a16887d5"). InnerVolumeSpecName "kube-api-access-7l7qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:50.830543 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:50.830518    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7qn\" (UniqueName: \"kubernetes.io/projected/2c2dfa3e-cbda-42e2-81c3-66f7a16887d5-kube-api-access-7l7qn\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:39:51.697584 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.697544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb" event={"ID":"88c3f4d7-a5cc-44f7-9f4d-605770334df0","Type":"ContainerStarted","Data":"f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6"}
Sep 08 13:39:51.699095 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.699069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj" event={"ID":"f1a9e050-e130-4f2d-aad9-58c8d13bf17d","Type":"ContainerStarted","Data":"bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c"}
Sep 08 13:39:51.699231 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.699094    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj" podUID="f1a9e050-e130-4f2d-aad9-58c8d13bf17d" containerName="authorino" containerID="cri-o://bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c" gracePeriod=30
Sep 08 13:39:51.699231 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.699118    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69554c6566-xdwcg"
Sep 08 13:39:51.717247 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.717197    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb" podStartSLOduration=1.207216082 podStartE2EDuration="1.717180844s" totalImagesPullingTime="509.964762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:39:51.71610993 +0000 UTC m=+700.067095892" watchObservedRunningTime="2026-09-08 13:39:51.717180844 +0000 UTC m=+700.068166848"
Sep 08 13:39:51.733453 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.733408    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj" podStartSLOduration=2.046586261 podStartE2EDuration="2.73337306s" totalImagesPullingTime="686.786799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:39:51.731850093 +0000 UTC m=+700.082836041" watchObservedRunningTime="2026-09-08 13:39:51.73337306 +0000 UTC m=+700.084359007"
Sep 08 13:39:51.747882 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.747849    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gtqft"]
Sep 08 13:39:51.748073 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.748051    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-gtqft" podUID="f3a653ff-3c6c-47b7-b09c-ed8b2790767e" containerName="authorino" containerID="cri-o://26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4" gracePeriod=30
Sep 08 13:39:51.761590 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.761558    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69554c6566-xdwcg"]
Sep 08 13:39:51.764990 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.764967    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69554c6566-xdwcg"]
Sep 08 13:39:51.997665 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:51.997644    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj"
Sep 08 13:39:52.000062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.000045    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-gtqft"
Sep 08 13:39:52.040231 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.040196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvqdn\" (UniqueName: \"kubernetes.io/projected/f3a653ff-3c6c-47b7-b09c-ed8b2790767e-kube-api-access-fvqdn\") pod \"f3a653ff-3c6c-47b7-b09c-ed8b2790767e\" (UID: \"f3a653ff-3c6c-47b7-b09c-ed8b2790767e\") "
Sep 08 13:39:52.040419 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.040256    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2qph\" (UniqueName: \"kubernetes.io/projected/f1a9e050-e130-4f2d-aad9-58c8d13bf17d-kube-api-access-b2qph\") pod \"f1a9e050-e130-4f2d-aad9-58c8d13bf17d\" (UID: \"f1a9e050-e130-4f2d-aad9-58c8d13bf17d\") "
Sep 08 13:39:52.042276 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.042241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a653ff-3c6c-47b7-b09c-ed8b2790767e-kube-api-access-fvqdn" (OuterVolumeSpecName: "kube-api-access-fvqdn") pod "f3a653ff-3c6c-47b7-b09c-ed8b2790767e" (UID: "f3a653ff-3c6c-47b7-b09c-ed8b2790767e"). InnerVolumeSpecName "kube-api-access-fvqdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:52.042276 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.042266    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a9e050-e130-4f2d-aad9-58c8d13bf17d-kube-api-access-b2qph" (OuterVolumeSpecName: "kube-api-access-b2qph") pod "f1a9e050-e130-4f2d-aad9-58c8d13bf17d" (UID: "f1a9e050-e130-4f2d-aad9-58c8d13bf17d"). InnerVolumeSpecName "kube-api-access-b2qph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:52.140899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.140860    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqdn\" (UniqueName: \"kubernetes.io/projected/f3a653ff-3c6c-47b7-b09c-ed8b2790767e-kube-api-access-fvqdn\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:39:52.140899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.140891    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qph\" (UniqueName: \"kubernetes.io/projected/f1a9e050-e130-4f2d-aad9-58c8d13bf17d-kube-api-access-b2qph\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:39:52.244019 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.243940    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2dfa3e-cbda-42e2-81c3-66f7a16887d5" path="/var/lib/kubelet/pods/2c2dfa3e-cbda-42e2-81c3-66f7a16887d5/volumes"
Sep 08 13:39:52.703872 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.703834    2561 generic.go:356] "Generic (PLEG): container finished" podID="f1a9e050-e130-4f2d-aad9-58c8d13bf17d" containerID="bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c" exitCode=0
Sep 08 13:39:52.704300 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.703897    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj"
Sep 08 13:39:52.704300 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.703921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj" event={"ID":"f1a9e050-e130-4f2d-aad9-58c8d13bf17d","Type":"ContainerDied","Data":"bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c"}
Sep 08 13:39:52.704300 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.703969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-vszdj" event={"ID":"f1a9e050-e130-4f2d-aad9-58c8d13bf17d","Type":"ContainerDied","Data":"e7a5925f66878767302d65c3549fc375a9ddc9738d36a0c89839eccfe4e365e6"}
Sep 08 13:39:52.704300 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.703993    2561 scope.go:117] "RemoveContainer" containerID="bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c"
Sep 08 13:39:52.705395 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.705355    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3a653ff-3c6c-47b7-b09c-ed8b2790767e" containerID="26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4" exitCode=0
Sep 08 13:39:52.705514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.705409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-gtqft" event={"ID":"f3a653ff-3c6c-47b7-b09c-ed8b2790767e","Type":"ContainerDied","Data":"26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4"}
Sep 08 13:39:52.705514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.705429    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-gtqft"
Sep 08 13:39:52.705514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.705449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-gtqft" event={"ID":"f3a653ff-3c6c-47b7-b09c-ed8b2790767e","Type":"ContainerDied","Data":"8972a696a434e3820487738868b7beefd24e5cf14ac34b444a9d7559dacadb06"}
Sep 08 13:39:52.712826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.712807    2561 scope.go:117] "RemoveContainer" containerID="bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c"
Sep 08 13:39:52.713086 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:39:52.713070    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c\": container with ID starting with bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c not found: ID does not exist" containerID="bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c"
Sep 08 13:39:52.713142 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.713093    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c"} err="failed to get container status \"bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c\": rpc error: code = NotFound desc = could not find container \"bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c\": container with ID starting with bfcca18ec81e184f74b6424d23d0205b5dc9df026f71f9e6630df1631b63457c not found: ID does not exist"
Sep 08 13:39:52.713142 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.713108    2561 scope.go:117] "RemoveContainer" containerID="26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4"
Sep 08 13:39:52.720754 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.720736    2561 scope.go:117] "RemoveContainer" containerID="26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4"
Sep 08 13:39:52.721044 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:39:52.721024    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4\": container with ID starting with 26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4 not found: ID does not exist" containerID="26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4"
Sep 08 13:39:52.721093 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.721053    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4"} err="failed to get container status \"26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4\": rpc error: code = NotFound desc = could not find container \"26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4\": container with ID starting with 26978c38737bd84045b522f331c77ce5599bf820721b5886b441b025bfc348b4 not found: ID does not exist"
Sep 08 13:39:52.725202 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.725178    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-vszdj"]
Sep 08 13:39:52.729573 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.729552    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-vszdj"]
Sep 08 13:39:52.742203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.742182    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gtqft"]
Sep 08 13:39:52.748670 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:52.748650    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gtqft"]
Sep 08 13:39:54.244037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:54.244004    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a9e050-e130-4f2d-aad9-58c8d13bf17d" path="/var/lib/kubelet/pods/f1a9e050-e130-4f2d-aad9-58c8d13bf17d/volumes"
Sep 08 13:39:54.244494 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:39:54.244306    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a653ff-3c6c-47b7-b09c-ed8b2790767e" path="/var/lib/kubelet/pods/f3a653ff-3c6c-47b7-b09c-ed8b2790767e/volumes"
Sep 08 13:41:23.642319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.642285    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"]
Sep 08 13:41:23.642793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.642580    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a9e050-e130-4f2d-aad9-58c8d13bf17d" containerName="authorino"
Sep 08 13:41:23.642793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.642592    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a9e050-e130-4f2d-aad9-58c8d13bf17d" containerName="authorino"
Sep 08 13:41:23.642793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.642600    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a653ff-3c6c-47b7-b09c-ed8b2790767e" containerName="authorino"
Sep 08 13:41:23.642793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.642605    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a653ff-3c6c-47b7-b09c-ed8b2790767e" containerName="authorino"
Sep 08 13:41:23.642793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.642653    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a653ff-3c6c-47b7-b09c-ed8b2790767e" containerName="authorino"
Sep 08 13:41:23.642793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.642661    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a9e050-e130-4f2d-aad9-58c8d13bf17d" containerName="authorino"
Sep 08 13:41:23.645426 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.645403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.649389 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.649358    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ws2qv\""
Sep 08 13:41:23.649389 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.649368    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 13:41:23.649551 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.649403    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 13:41:23.649551 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.649405    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 08 13:41:23.661240 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.661215    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"]
Sep 08 13:41:23.718667 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.718635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87ff\" (UniqueName: \"kubernetes.io/projected/cad7f373-91df-408b-8fac-4a0a889f5b05-kube-api-access-b87ff\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.718667 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.718671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad7f373-91df-408b-8fac-4a0a889f5b05-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.718878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.718696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.718878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.718758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.718878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.718832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.718878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.718850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820029 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.819996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820029 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.820030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820243 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.820079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87ff\" (UniqueName: \"kubernetes.io/projected/cad7f373-91df-408b-8fac-4a0a889f5b05-kube-api-access-b87ff\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820243 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.820108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad7f373-91df-408b-8fac-4a0a889f5b05-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820243 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.820139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820243 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.820160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820542 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.820517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820606 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.820538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.820606 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.820570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.822286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.822266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cad7f373-91df-408b-8fac-4a0a889f5b05-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.822567 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.822550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad7f373-91df-408b-8fac-4a0a889f5b05-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.829957 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.829930    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87ff\" (UniqueName: \"kubernetes.io/projected/cad7f373-91df-408b-8fac-4a0a889f5b05-kube-api-access-b87ff\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnxsx\" (UID: \"cad7f373-91df-408b-8fac-4a0a889f5b05\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:23.955489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:23.955454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:24.229763 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:24.229674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"]
Sep 08 13:41:24.232482 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:41:24.232455    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad7f373_91df_408b_8fac_4a0a889f5b05.slice/crio-93ccf96f9265c9dbe105bdc30a733aab3980936c0a638b64881ab94ad36fe08c WatchSource:0}: Error finding container 93ccf96f9265c9dbe105bdc30a733aab3980936c0a638b64881ab94ad36fe08c: Status 404 returned error can't find the container with id 93ccf96f9265c9dbe105bdc30a733aab3980936c0a638b64881ab94ad36fe08c
Sep 08 13:41:24.233843 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:24.233821    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:41:25.065329 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:25.065299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx" event={"ID":"cad7f373-91df-408b-8fac-4a0a889f5b05","Type":"ContainerStarted","Data":"93ccf96f9265c9dbe105bdc30a733aab3980936c0a638b64881ab94ad36fe08c"}
Sep 08 13:41:26.070697 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:26.070657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx" event={"ID":"cad7f373-91df-408b-8fac-4a0a889f5b05","Type":"ContainerStarted","Data":"e68992feb598fe248e4e7e13c1300b828b28be3d174ec174c8c128441fafab8a"}
Sep 08 13:41:26.091130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:26.091073    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx" podStartSLOduration=1.9145795620000001 podStartE2EDuration="3.091058527s" totalImagesPullingTime="1.176478965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:26.089626885 +0000 UTC m=+794.440612833" watchObservedRunningTime="2026-09-08 13:41:26.091058527 +0000 UTC m=+794.442044475"
Sep 08 13:41:33.956466 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:33.956419    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:33.957069 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:33.956526    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:33.968908 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:33.968879    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:41:34.111838 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:41:34.111806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnxsx"
Sep 08 13:42:42.914142 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.914108    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"]
Sep 08 13:42:42.917270 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.917249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:42.920222 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.920203    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 13:42:42.927661 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.927639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:42.927745 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.927667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:42.927745 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.927702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:42.927745 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.927725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e197378-390c-46bc-a252-2a62ff4a4999-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:42.927745 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.927741    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxwp\" (UniqueName: \"kubernetes.io/projected/1e197378-390c-46bc-a252-2a62ff4a4999-kube-api-access-tzxwp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:42.927871 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.927759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:42.931867 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:42.931846    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"]
Sep 08 13:42:43.029127 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.029127 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.029326 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.029326 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e197378-390c-46bc-a252-2a62ff4a4999-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.029326 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxwp\" (UniqueName: \"kubernetes.io/projected/1e197378-390c-46bc-a252-2a62ff4a4999-kube-api-access-tzxwp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.029483 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.029624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.029675 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.029734 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.029686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.031530 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.031503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e197378-390c-46bc-a252-2a62ff4a4999-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.031824 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.031805    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e197378-390c-46bc-a252-2a62ff4a4999-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.038077 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.038055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxwp\" (UniqueName: \"kubernetes.io/projected/1e197378-390c-46bc-a252-2a62ff4a4999-kube-api-access-tzxwp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-98ggc\" (UID: \"1e197378-390c-46bc-a252-2a62ff4a4999\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.226032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.226000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:43.497837 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:43.497813    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"]
Sep 08 13:42:43.500476 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:42:43.500447    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e197378_390c_46bc_a252_2a62ff4a4999.slice/crio-8abdbacc94a437131418d82a5b3b882ac997d554d02664ed16ee78dc5702a18b WatchSource:0}: Error finding container 8abdbacc94a437131418d82a5b3b882ac997d554d02664ed16ee78dc5702a18b: Status 404 returned error can't find the container with id 8abdbacc94a437131418d82a5b3b882ac997d554d02664ed16ee78dc5702a18b
Sep 08 13:42:44.392957 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:44.392921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc" event={"ID":"1e197378-390c-46bc-a252-2a62ff4a4999","Type":"ContainerStarted","Data":"4564910609a39f6ccbb9d0babcec21b3fb348bac8052c909e63a5fbc31ebd17d"}
Sep 08 13:42:44.393471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:44.392974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc" event={"ID":"1e197378-390c-46bc-a252-2a62ff4a4999","Type":"ContainerStarted","Data":"8abdbacc94a437131418d82a5b3b882ac997d554d02664ed16ee78dc5702a18b"}
Sep 08 13:42:44.412972 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:44.412926    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc" podStartSLOduration=2.204101612 podStartE2EDuration="2.412909851s" totalImagesPullingTime="208.808239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:42:44.41166609 +0000 UTC m=+872.762652041" watchObservedRunningTime="2026-09-08 13:42:44.412909851 +0000 UTC m=+872.763895799"
Sep 08 13:42:53.226712 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:53.226665    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:53.226712 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:53.226725    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:53.239836 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:53.239805    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:42:53.443727 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:42:53.443692    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-98ggc"
Sep 08 13:43:35.732597 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.732548    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dcccdf9cc-zw28h"]
Sep 08 13:43:35.735818 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.735800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dcccdf9cc-zw28h"
Sep 08 13:43:35.743307 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.743283    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dcccdf9cc-zw28h"]
Sep 08 13:43:35.844450 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.844412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8567f5f6-08b5-43ea-b35e-9b77dea07c42-tls-cert\") pod \"authorino-6dcccdf9cc-zw28h\" (UID: \"8567f5f6-08b5-43ea-b35e-9b77dea07c42\") " pod="kuadrant-system/authorino-6dcccdf9cc-zw28h"
Sep 08 13:43:35.844618 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.844478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccbm\" (UniqueName: \"kubernetes.io/projected/8567f5f6-08b5-43ea-b35e-9b77dea07c42-kube-api-access-6ccbm\") pod \"authorino-6dcccdf9cc-zw28h\" (UID: \"8567f5f6-08b5-43ea-b35e-9b77dea07c42\") " pod="kuadrant-system/authorino-6dcccdf9cc-zw28h"
Sep 08 13:43:35.945676 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.945642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8567f5f6-08b5-43ea-b35e-9b77dea07c42-tls-cert\") pod \"authorino-6dcccdf9cc-zw28h\" (UID: \"8567f5f6-08b5-43ea-b35e-9b77dea07c42\") " pod="kuadrant-system/authorino-6dcccdf9cc-zw28h"
Sep 08 13:43:35.945843 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.945691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccbm\" (UniqueName: \"kubernetes.io/projected/8567f5f6-08b5-43ea-b35e-9b77dea07c42-kube-api-access-6ccbm\") pod \"authorino-6dcccdf9cc-zw28h\" (UID: \"8567f5f6-08b5-43ea-b35e-9b77dea07c42\") " pod="kuadrant-system/authorino-6dcccdf9cc-zw28h"
Sep 08 13:43:35.948046 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.948028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8567f5f6-08b5-43ea-b35e-9b77dea07c42-tls-cert\") pod \"authorino-6dcccdf9cc-zw28h\" (UID: \"8567f5f6-08b5-43ea-b35e-9b77dea07c42\") " pod="kuadrant-system/authorino-6dcccdf9cc-zw28h"
Sep 08 13:43:35.953858 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:35.953833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccbm\" (UniqueName: \"kubernetes.io/projected/8567f5f6-08b5-43ea-b35e-9b77dea07c42-kube-api-access-6ccbm\") pod \"authorino-6dcccdf9cc-zw28h\" (UID: \"8567f5f6-08b5-43ea-b35e-9b77dea07c42\") " pod="kuadrant-system/authorino-6dcccdf9cc-zw28h"
Sep 08 13:43:36.044830 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:36.044744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dcccdf9cc-zw28h"
Sep 08 13:43:36.317996 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:36.316294    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dcccdf9cc-zw28h"]
Sep 08 13:43:36.319696 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:43:36.319663    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8567f5f6_08b5_43ea_b35e_9b77dea07c42.slice/crio-85995b0e8f48a8dd7e9f64ea3ca2b671c129190c7617ace55060f2f35316a7de WatchSource:0}: Error finding container 85995b0e8f48a8dd7e9f64ea3ca2b671c129190c7617ace55060f2f35316a7de: Status 404 returned error can't find the container with id 85995b0e8f48a8dd7e9f64ea3ca2b671c129190c7617ace55060f2f35316a7de
Sep 08 13:43:36.603146 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:36.603063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dcccdf9cc-zw28h" event={"ID":"8567f5f6-08b5-43ea-b35e-9b77dea07c42","Type":"ContainerStarted","Data":"85995b0e8f48a8dd7e9f64ea3ca2b671c129190c7617ace55060f2f35316a7de"}
Sep 08 13:43:37.609231 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:37.609183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dcccdf9cc-zw28h" event={"ID":"8567f5f6-08b5-43ea-b35e-9b77dea07c42","Type":"ContainerStarted","Data":"2ecf20e11cc5c1baf7347b42a23e29c7c575d864aee5031fa5d9edf7faa70501"}
Sep 08 13:43:37.653083 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:37.653022    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6dcccdf9cc-zw28h" podStartSLOduration=1.942280191 podStartE2EDuration="2.652996944s" totalImagesPullingTime="710.716753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:43:37.62793612 +0000 UTC m=+925.978922068" watchObservedRunningTime="2026-09-08 13:43:37.652996944 +0000 UTC m=+926.003982893"
Sep 08 13:43:37.653846 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:37.653821    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f64b7bcfb-2s7lb"]
Sep 08 13:43:37.654100 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:37.654065    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb" podUID="88c3f4d7-a5cc-44f7-9f4d-605770334df0" containerName="authorino" containerID="cri-o://f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6" gracePeriod=30
Sep 08 13:43:37.907710 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:37.907682    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:43:38.060134 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.060097    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4p8b\" (UniqueName: \"kubernetes.io/projected/88c3f4d7-a5cc-44f7-9f4d-605770334df0-kube-api-access-p4p8b\") pod \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\" (UID: \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\") "
Sep 08 13:43:38.060319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.060154    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88c3f4d7-a5cc-44f7-9f4d-605770334df0-tls-cert\") pod \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\" (UID: \"88c3f4d7-a5cc-44f7-9f4d-605770334df0\") "
Sep 08 13:43:38.062356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.062313    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c3f4d7-a5cc-44f7-9f4d-605770334df0-kube-api-access-p4p8b" (OuterVolumeSpecName: "kube-api-access-p4p8b") pod "88c3f4d7-a5cc-44f7-9f4d-605770334df0" (UID: "88c3f4d7-a5cc-44f7-9f4d-605770334df0"). InnerVolumeSpecName "kube-api-access-p4p8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:43:38.069814 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.069789    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c3f4d7-a5cc-44f7-9f4d-605770334df0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "88c3f4d7-a5cc-44f7-9f4d-605770334df0" (UID: "88c3f4d7-a5cc-44f7-9f4d-605770334df0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:43:38.161428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.161338    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/88c3f4d7-a5cc-44f7-9f4d-605770334df0-tls-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:43:38.161428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.161368    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4p8b\" (UniqueName: \"kubernetes.io/projected/88c3f4d7-a5cc-44f7-9f4d-605770334df0-kube-api-access-p4p8b\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:43:38.614926 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.614889    2561 generic.go:356] "Generic (PLEG): container finished" podID="88c3f4d7-a5cc-44f7-9f4d-605770334df0" containerID="f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6" exitCode=0
Sep 08 13:43:38.615434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.614936    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb"
Sep 08 13:43:38.615434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.614927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb" event={"ID":"88c3f4d7-a5cc-44f7-9f4d-605770334df0","Type":"ContainerDied","Data":"f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6"}
Sep 08 13:43:38.615434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.615049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f64b7bcfb-2s7lb" event={"ID":"88c3f4d7-a5cc-44f7-9f4d-605770334df0","Type":"ContainerDied","Data":"392d2fd14f92898497df7acf9021050bbf832d885a0c5843e0b0d05f123e4ee4"}
Sep 08 13:43:38.615434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.615072    2561 scope.go:117] "RemoveContainer" containerID="f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6"
Sep 08 13:43:38.623165 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.623144    2561 scope.go:117] "RemoveContainer" containerID="f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6"
Sep 08 13:43:38.623452 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:43:38.623429    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6\": container with ID starting with f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6 not found: ID does not exist" containerID="f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6"
Sep 08 13:43:38.623573 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.623458    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6"} err="failed to get container status \"f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6\": rpc error: code = NotFound desc = could not find container \"f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6\": container with ID starting with f515969965258e7fdc14248a898cf29f628b8be8e5ddf0da09a46f8d8f37dbf6 not found: ID does not exist"
Sep 08 13:43:38.635902 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.635876    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f64b7bcfb-2s7lb"]
Sep 08 13:43:38.643744 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:38.643715    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f64b7bcfb-2s7lb"]
Sep 08 13:43:40.245698 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:40.245662    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c3f4d7-a5cc-44f7-9f4d-605770334df0" path="/var/lib/kubelet/pods/88c3f4d7-a5cc-44f7-9f4d-605770334df0/volumes"
Sep 08 13:43:58.765902 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.765867    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7729-140660209793712"]
Sep 08 13:43:58.766439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.766175    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c3f4d7-a5cc-44f7-9f4d-605770334df0" containerName="authorino"
Sep 08 13:43:58.766439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.766188    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c3f4d7-a5cc-44f7-9f4d-605770334df0" containerName="authorino"
Sep 08 13:43:58.766439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.766230    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c3f4d7-a5cc-44f7-9f4d-605770334df0" containerName="authorino"
Sep 08 13:43:58.769292 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.769276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-140660209793712"
Sep 08 13:43:58.773186 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.773169    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qzs56\""
Sep 08 13:43:58.779879 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.779858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7729-140660209793712"]
Sep 08 13:43:58.820971 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.820943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmg85\" (UniqueName: \"kubernetes.io/projected/8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c-kube-api-access-qmg85\") pod \"test-curl-7729-140660209793712\" (UID: \"8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c\") " pod="opendatahub/test-curl-7729-140660209793712"
Sep 08 13:43:58.922307 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.922271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmg85\" (UniqueName: \"kubernetes.io/projected/8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c-kube-api-access-qmg85\") pod \"test-curl-7729-140660209793712\" (UID: \"8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c\") " pod="opendatahub/test-curl-7729-140660209793712"
Sep 08 13:43:58.931846 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:58.931824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmg85\" (UniqueName: \"kubernetes.io/projected/8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c-kube-api-access-qmg85\") pod \"test-curl-7729-140660209793712\" (UID: \"8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c\") " pod="opendatahub/test-curl-7729-140660209793712"
Sep 08 13:43:59.078308 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:59.078221    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-140660209793712"
Sep 08 13:43:59.344632 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:59.344609    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7729-140660209793712"]
Sep 08 13:43:59.710185 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:43:59.710150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7729-140660209793712" event={"ID":"8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c","Type":"ContainerStarted","Data":"ace5b96777ae5613dc73e6a22e35b4e417bbe648598ba38510f292dbf6b3dd76"}
Sep 08 13:44:02.726267 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:02.726232    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c" containerID="daed93cd5f1a5ee5588f8937febf7083201a586a16d0aa5e42fc90b82f92cdad" exitCode=0
Sep 08 13:44:02.726700 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:02.726316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7729-140660209793712" event={"ID":"8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c","Type":"ContainerDied","Data":"daed93cd5f1a5ee5588f8937febf7083201a586a16d0aa5e42fc90b82f92cdad"}
Sep 08 13:44:03.853555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:03.853532    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-140660209793712"
Sep 08 13:44:03.860447 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:03.860427    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmg85\" (UniqueName: \"kubernetes.io/projected/8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c-kube-api-access-qmg85\") pod \"8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c\" (UID: \"8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c\") "
Sep 08 13:44:03.862467 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:03.862440    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c-kube-api-access-qmg85" (OuterVolumeSpecName: "kube-api-access-qmg85") pod "8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c" (UID: "8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c"). InnerVolumeSpecName "kube-api-access-qmg85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:44:03.913255 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:03.913232    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7729-140660209793712_8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c/test-curl-7729-140660209793712/0.log"
Sep 08 13:44:03.960986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:03.960951    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmg85\" (UniqueName: \"kubernetes.io/projected/8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c-kube-api-access-qmg85\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:03.962967 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:03.962942    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7729-140660209793712"]
Sep 08 13:44:03.970821 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:03.970798    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7729-140660209793712"]
Sep 08 13:44:04.099857 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.099786    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7729-140660209793968"]
Sep 08 13:44:04.100069 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.100057    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c" containerName="test-curl-7729-140660209793712"
Sep 08 13:44:04.100157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.100070    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c" containerName="test-curl-7729-140660209793712"
Sep 08 13:44:04.100157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.100138    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c" containerName="test-curl-7729-140660209793712"
Sep 08 13:44:04.102862 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.102844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:04.110112 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.110093    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7729-140660209793968"]
Sep 08 13:44:04.162500 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.162469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pt2m\" (UniqueName: \"kubernetes.io/projected/17dd48df-84e3-4a68-834f-7524aa2407f3-kube-api-access-2pt2m\") pod \"test-curl-7729-140660209793968\" (UID: \"17dd48df-84e3-4a68-834f-7524aa2407f3\") " pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:04.244934 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.244894    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c" path="/var/lib/kubelet/pods/8a7c1f85-bfe9-4d6f-87a4-cee6b265c98c/volumes"
Sep 08 13:44:04.262920 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.262899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pt2m\" (UniqueName: \"kubernetes.io/projected/17dd48df-84e3-4a68-834f-7524aa2407f3-kube-api-access-2pt2m\") pod \"test-curl-7729-140660209793968\" (UID: \"17dd48df-84e3-4a68-834f-7524aa2407f3\") " pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:04.293354 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.293332    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pt2m\" (UniqueName: \"kubernetes.io/projected/17dd48df-84e3-4a68-834f-7524aa2407f3-kube-api-access-2pt2m\") pod \"test-curl-7729-140660209793968\" (UID: \"17dd48df-84e3-4a68-834f-7524aa2407f3\") " pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:04.413042 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.412938    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:04.684743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.684719    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7729-140660209793968"]
Sep 08 13:44:04.687020 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:04.686994    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17dd48df_84e3_4a68_834f_7524aa2407f3.slice/crio-5a01b8db07743225f457af867f11bf90c78a77e9f59e41d874b0c53ad087ec2d WatchSource:0}: Error finding container 5a01b8db07743225f457af867f11bf90c78a77e9f59e41d874b0c53ad087ec2d: Status 404 returned error can't find the container with id 5a01b8db07743225f457af867f11bf90c78a77e9f59e41d874b0c53ad087ec2d
Sep 08 13:44:04.736321 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.736279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7729-140660209793968" event={"ID":"17dd48df-84e3-4a68-834f-7524aa2407f3","Type":"ContainerStarted","Data":"5a01b8db07743225f457af867f11bf90c78a77e9f59e41d874b0c53ad087ec2d"}
Sep 08 13:44:04.737498 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.737481    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-140660209793712"
Sep 08 13:44:04.737597 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:04.737483    2561 scope.go:117] "RemoveContainer" containerID="daed93cd5f1a5ee5588f8937febf7083201a586a16d0aa5e42fc90b82f92cdad"
Sep 08 13:44:05.743683 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:05.743647    2561 generic.go:356] "Generic (PLEG): container finished" podID="17dd48df-84e3-4a68-834f-7524aa2407f3" containerID="726da0d110b40b8a6503ad4f3a0c1d21b55481b81b5f6257f4592528edab8d29" exitCode=0
Sep 08 13:44:05.744047 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:05.743703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7729-140660209793968" event={"ID":"17dd48df-84e3-4a68-834f-7524aa2407f3","Type":"ContainerDied","Data":"726da0d110b40b8a6503ad4f3a0c1d21b55481b81b5f6257f4592528edab8d29"}
Sep 08 13:44:06.492010 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.491971    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"]
Sep 08 13:44:06.495194 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.495169    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.498748 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.498726    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0b415db2-openshift-default-dockercfg-mjxjz\""
Sep 08 13:44:06.499076 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.499059    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 13:44:06.516176 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.516148    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"]
Sep 08 13:44:06.584899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.584862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-certs\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.585075 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.584921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-envoy\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.585075 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.584950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqz8w\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-kube-api-access-cqz8w\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.585075 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.585003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-token\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.585075 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.585044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-socket\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.585240 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.585075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-podinfo\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.585240 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.585103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-data\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.585240 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.585130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-credential-socket\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.585240 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.585190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istiod-ca-cert\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686014 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.685974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-token\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-socket\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-podinfo\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-data\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-credential-socket\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istiod-ca-cert\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-certs\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-envoy\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686585 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-socket\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686585 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-data\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686585 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqz8w\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-kube-api-access-cqz8w\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686806 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-credential-socket\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686806 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-certs\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.686907 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.686859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istiod-ca-cert\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.688623 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.688599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-envoy\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.688806 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.688786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-podinfo\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.695342 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.695319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqz8w\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-kube-api-access-cqz8w\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.695477 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.695340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-token\") pod \"e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") " pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.804672 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.804642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:06.875290 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.875268    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:06.888331 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.888309    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pt2m\" (UniqueName: \"kubernetes.io/projected/17dd48df-84e3-4a68-834f-7524aa2407f3-kube-api-access-2pt2m\") pod \"17dd48df-84e3-4a68-834f-7524aa2407f3\" (UID: \"17dd48df-84e3-4a68-834f-7524aa2407f3\") "
Sep 08 13:44:06.890340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.890297    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dd48df-84e3-4a68-834f-7524aa2407f3-kube-api-access-2pt2m" (OuterVolumeSpecName: "kube-api-access-2pt2m") pod "17dd48df-84e3-4a68-834f-7524aa2407f3" (UID: "17dd48df-84e3-4a68-834f-7524aa2407f3"). InnerVolumeSpecName "kube-api-access-2pt2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:44:06.896821 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.896798    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7729-140660209793968_17dd48df-84e3-4a68-834f-7524aa2407f3/test-curl-7729-140660209793968/0.log"
Sep 08 13:44:06.919474 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.919449    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7729-140660209793968"]
Sep 08 13:44:06.925287 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.925265    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7729-140660209793968"]
Sep 08 13:44:06.989152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:06.989114    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pt2m\" (UniqueName: \"kubernetes.io/projected/17dd48df-84e3-4a68-834f-7524aa2407f3-kube-api-access-2pt2m\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:07.085950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.085925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"]
Sep 08 13:44:07.088472 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:07.088446    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a0eee6_0d1f_41f7_9f13_6de3b476be44.slice/crio-9ab4031e0660b572e515df16c0c72f795537be5b7db17e534539f934f4250209 WatchSource:0}: Error finding container 9ab4031e0660b572e515df16c0c72f795537be5b7db17e534539f934f4250209: Status 404 returned error can't find the container with id 9ab4031e0660b572e515df16c0c72f795537be5b7db17e534539f934f4250209
Sep 08 13:44:07.754730 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.754684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" event={"ID":"86a0eee6-0d1f-41f7-9f13-6de3b476be44","Type":"ContainerStarted","Data":"9ab4031e0660b572e515df16c0c72f795537be5b7db17e534539f934f4250209"}
Sep 08 13:44:07.756098 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.756071    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a01b8db07743225f457af867f11bf90c78a77e9f59e41d874b0c53ad087ec2d"
Sep 08 13:44:07.756211 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.756144    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:07.768482 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:44:07.768446    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7729-140660209793968\" is forbidden: User \"system:node:ip-10-0-130-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-208.ec2.internal' and this object" podUID="17dd48df-84e3-4a68-834f-7524aa2407f3" pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:07.826939 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.826909    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"]
Sep 08 13:44:07.827311 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.827230    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dd48df-84e3-4a68-834f-7524aa2407f3" containerName="test-curl-7729-140660209793968"
Sep 08 13:44:07.827311 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.827248    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dd48df-84e3-4a68-834f-7524aa2407f3" containerName="test-curl-7729-140660209793968"
Sep 08 13:44:07.827450 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.827317    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17dd48df-84e3-4a68-834f-7524aa2407f3" containerName="test-curl-7729-140660209793968"
Sep 08 13:44:07.830163 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.830138    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.833537 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:44:07.833369    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7729-140660209793968\" is forbidden: User \"system:node:ip-10-0-130-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-208.ec2.internal' and this object" podUID="17dd48df-84e3-4a68-834f-7524aa2407f3" pod="opendatahub/test-curl-7729-140660209793968"
Sep 08 13:44:07.833678 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.833661    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b719d2-openshift-default-dockercfg-2mkcm\""
Sep 08 13:44:07.867822 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.867787    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"]
Sep 08 13:44:07.896094 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-envoy\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.896094 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-token\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.896309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-socket\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.896309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-data\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.896309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-credential-socket\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.896309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26bcd53-9ee2-4082-809b-b28852380d9a-istiod-ca-cert\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.896553 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-podinfo\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.896553 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-certs\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.896553 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.896482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrlhl\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-kube-api-access-hrlhl\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997168 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-certs\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997168 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlhl\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-kube-api-access-hrlhl\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997480 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-envoy\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997480 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-token\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-socket\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-certs\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997582    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-data\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-credential-socket\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26bcd53-9ee2-4082-809b-b28852380d9a-istiod-ca-cert\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.997688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.997681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-podinfo\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.998268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.998144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-data\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.998268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.998171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-credential-socket\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.998268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.998232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-socket\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:07.998485 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:07.998459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26bcd53-9ee2-4082-809b-b28852380d9a-istiod-ca-cert\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:08.000212 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:08.000190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-envoy\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:08.000459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:08.000442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-podinfo\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:08.015080 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:08.015047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-token\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:08.015188 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:08.015153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlhl\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-kube-api-access-hrlhl\") pod \"e2e-autores-b719d2-openshift-default-65496b95cb-b8prg\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") " pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:08.141685 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:08.141650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:08.247036 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:08.247000    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17dd48df-84e3-4a68-834f-7524aa2407f3" path="/var/lib/kubelet/pods/17dd48df-84e3-4a68-834f-7524aa2407f3/volumes"
Sep 08 13:44:08.428714 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:08.428685    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"]
Sep 08 13:44:08.430478 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:08.430445    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26bcd53_9ee2_4082_809b_b28852380d9a.slice/crio-77cc13750885408fa2c164c515ee1d3ce1b0c6f496c13ac0272599ea1f027749 WatchSource:0}: Error finding container 77cc13750885408fa2c164c515ee1d3ce1b0c6f496c13ac0272599ea1f027749: Status 404 returned error can't find the container with id 77cc13750885408fa2c164c515ee1d3ce1b0c6f496c13ac0272599ea1f027749
Sep 08 13:44:08.762306 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:08.762226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" event={"ID":"c26bcd53-9ee2-4082-809b-b28852380d9a","Type":"ContainerStarted","Data":"77cc13750885408fa2c164c515ee1d3ce1b0c6f496c13ac0272599ea1f027749"}
Sep 08 13:44:09.630819 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.630779    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:09.631120 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.630852    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:09.631120 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.630881    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:09.631727 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.631691    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:09.631862 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.631839    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:09.631953 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.631898    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:09.770705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.770645    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" event={"ID":"c26bcd53-9ee2-4082-809b-b28852380d9a","Type":"ContainerStarted","Data":"da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7"}
Sep 08 13:44:09.772648 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.772610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" event={"ID":"86a0eee6-0d1f-41f7-9f13-6de3b476be44","Type":"ContainerStarted","Data":"4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18"}
Sep 08 13:44:09.793726 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.793672    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" podStartSLOduration=1.595325197 podStartE2EDuration="2.793653958s" totalImagesPullingTime="1.198328761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:09.791523443 +0000 UTC m=+958.142509393" watchObservedRunningTime="2026-09-08 13:44:09.793653958 +0000 UTC m=+958.144639903"
Sep 08 13:44:09.805525 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.805493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:09.813905 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:09.813856    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" podStartSLOduration=1.273908293 podStartE2EDuration="3.813840012s" totalImagesPullingTime="2.539931719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:09.812506871 +0000 UTC m=+958.163492815" watchObservedRunningTime="2026-09-08 13:44:09.813840012 +0000 UTC m=+958.164825959"
Sep 08 13:44:10.142563 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:10.142526    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:10.146992 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:10.146970    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:10.776979 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:10.776945    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:10.777719 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:10.777699    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:10.809651 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:10.809624    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:11.781674 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:11.781643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:11.782458 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:11.782440    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:13.344391 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.344349    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"]
Sep 08 13:44:13.348367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.348348    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:13.351703 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.351679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0b415db2-dockercfg-92zkr\""
Sep 08 13:44:13.351855 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.351835    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0b415db2\""
Sep 08 13:44:13.366596 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.366574    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"]
Sep 08 13:44:13.445769 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.445738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88tw8\" (UniqueName: \"kubernetes.io/projected/d6336460-76de-4d47-9e19-af6ae23ccefb-kube-api-access-88tw8\") pod \"payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6\" (UID: \"d6336460-76de-4d47-9e19-af6ae23ccefb\") " pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:13.445924 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.445796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6336460-76de-4d47-9e19-af6ae23ccefb-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6\" (UID: \"d6336460-76de-4d47-9e19-af6ae23ccefb\") " pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:13.546343 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.546308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88tw8\" (UniqueName: \"kubernetes.io/projected/d6336460-76de-4d47-9e19-af6ae23ccefb-kube-api-access-88tw8\") pod \"payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6\" (UID: \"d6336460-76de-4d47-9e19-af6ae23ccefb\") " pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:13.546520 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.546355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6336460-76de-4d47-9e19-af6ae23ccefb-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6\" (UID: \"d6336460-76de-4d47-9e19-af6ae23ccefb\") " pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:13.547055 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.547037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6336460-76de-4d47-9e19-af6ae23ccefb-plugins-config-volume\") pod \"payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6\" (UID: \"d6336460-76de-4d47-9e19-af6ae23ccefb\") " pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:13.555459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.555431    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88tw8\" (UniqueName: \"kubernetes.io/projected/d6336460-76de-4d47-9e19-af6ae23ccefb-kube-api-access-88tw8\") pod \"payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6\" (UID: \"d6336460-76de-4d47-9e19-af6ae23ccefb\") " pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:13.658294 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.658218    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:13.944138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:13.944108    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"]
Sep 08 13:44:13.946744 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:13.946714    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6336460_76de_4d47_9e19_af6ae23ccefb.slice/crio-f55bb167e698cd5c45216d3e84c1d14a1ef01503e6a91282e4efd7052c3d3fac WatchSource:0}: Error finding container f55bb167e698cd5c45216d3e84c1d14a1ef01503e6a91282e4efd7052c3d3fac: Status 404 returned error can't find the container with id f55bb167e698cd5c45216d3e84c1d14a1ef01503e6a91282e4efd7052c3d3fac
Sep 08 13:44:14.796864 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:14.796815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6" event={"ID":"d6336460-76de-4d47-9e19-af6ae23ccefb","Type":"ContainerStarted","Data":"f55bb167e698cd5c45216d3e84c1d14a1ef01503e6a91282e4efd7052c3d3fac"}
Sep 08 13:44:15.809529 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:15.809487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6" event={"ID":"d6336460-76de-4d47-9e19-af6ae23ccefb","Type":"ContainerStarted","Data":"6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda"}
Sep 08 13:44:15.809930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:15.809656    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:15.835393 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:15.835329    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6" podStartSLOduration=1.117158354 podStartE2EDuration="2.835315143s" totalImagesPullingTime="1.718156789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:15.834368989 +0000 UTC m=+964.185354937" watchObservedRunningTime="2026-09-08 13:44:15.835315143 +0000 UTC m=+964.186301091"
Sep 08 13:44:26.783058 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.782982    2561 patch_prober.go:28] interesting pod/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Sep 08 13:44:26.783058 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.783038    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Sep 08 13:44:26.816357 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.816320    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:26.861164 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.861084    2561 generic.go:356] "Generic (PLEG): container finished" podID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerID="4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18" exitCode=0
Sep 08 13:44:26.861314 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.861167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" event={"ID":"86a0eee6-0d1f-41f7-9f13-6de3b476be44","Type":"ContainerDied","Data":"4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18"}
Sep 08 13:44:26.861686 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.861667    2561 scope.go:117] "RemoveContainer" containerID="4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18"
Sep 08 13:44:26.862989 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.862957    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:26.863072 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.863044    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:26.863114 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:26.863097    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:27.884624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:27.883876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" event={"ID":"86a0eee6-0d1f-41f7-9f13-6de3b476be44","Type":"ContainerStarted","Data":"baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3"}
Sep 08 13:44:28.806068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:28.806030    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:28.806282 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:28.806082    2561 patch_prober.go:28] interesting pod/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Sep 08 13:44:28.806282 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:28.806133    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Sep 08 13:44:29.164456 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.164217    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"]
Sep 08 13:44:29.171740 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.171713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.177401 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.176225    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a7ef4b-openshift-default-dockercfg-8k2nz\""
Sep 08 13:44:29.185960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.185936    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"]
Sep 08 13:44:29.298066 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-envoy\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.298066 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-certs\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.298317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istiod-ca-cert\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.298317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-token\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.298317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-socket\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.298317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-data\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.298317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-podinfo\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.298317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-credential-socket\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.298317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.298220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksbm\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-kube-api-access-hksbm\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399009 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.398967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-credential-socket\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399009 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksbm\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-kube-api-access-hksbm\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399208 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-envoy\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399208 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-certs\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399208 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istiod-ca-cert\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399208 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-token\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399208 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-socket\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399208 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-data\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399208 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-podinfo\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.399838 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-certs\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.400010 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.399990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-credential-socket\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.400646 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.400623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istiod-ca-cert\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.401025 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.400976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-socket\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.401135 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.401018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-data\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.401635 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.401591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-podinfo\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.401749 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.401702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-envoy\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.408693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.408649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-token\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.409035 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.409015    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksbm\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-kube-api-access-hksbm\") pod \"e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") " pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.500561 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.500518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:29.805787 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.805714    2561 patch_prober.go:28] interesting pod/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Sep 08 13:44:29.805787 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.805776    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Sep 08 13:44:29.829579 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.829511    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"]
Sep 08 13:44:29.831594 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:29.831570    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf004aa6_08ae_47f7_a6b3_c7468e6699f4.slice/crio-c974de5e1b6cb87496e403b6dda8fd8f604bce928872c92be43e84ba613b303e WatchSource:0}: Error finding container c974de5e1b6cb87496e403b6dda8fd8f604bce928872c92be43e84ba613b303e: Status 404 returned error can't find the container with id c974de5e1b6cb87496e403b6dda8fd8f604bce928872c92be43e84ba613b303e
Sep 08 13:44:29.833851 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.833819    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:29.833968 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.833890    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:29.833968 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.833931    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:29.896775 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:29.896744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" event={"ID":"df004aa6-08ae-47f7-a6b3-c7468e6699f4","Type":"ContainerStarted","Data":"c974de5e1b6cb87496e403b6dda8fd8f604bce928872c92be43e84ba613b303e"}
Sep 08 13:44:30.805397 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:30.805347    2561 patch_prober.go:28] interesting pod/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Sep 08 13:44:30.805779 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:30.805432    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Sep 08 13:44:30.902410 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:30.902365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" event={"ID":"df004aa6-08ae-47f7-a6b3-c7468e6699f4","Type":"ContainerStarted","Data":"ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5"}
Sep 08 13:44:30.928720 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:30.928666    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" podStartSLOduration=1.9286487719999998 podStartE2EDuration="1.928648772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:30.92625595 +0000 UTC m=+979.277241898" watchObservedRunningTime="2026-09-08 13:44:30.928648772 +0000 UTC m=+979.279634724"
Sep 08 13:44:31.500761 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:31.500722    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:31.505652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:31.505629    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:31.805700 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:31.805615    2561 patch_prober.go:28] interesting pod/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Sep 08 13:44:31.805700 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:31.805684    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Sep 08 13:44:31.906484 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:31.906458    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:31.907485 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:31.907471    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:32.805921 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:32.805876    2561 patch_prober.go:28] interesting pod/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body=
Sep 08 13:44:32.806442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:32.805939    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused"
Sep 08 13:44:33.810584 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:33.810493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:33.811033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:33.810880    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:33.811459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:33.811441    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:36.930755 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.930721    2561 generic.go:356] "Generic (PLEG): container finished" podID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerID="da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7" exitCode=0
Sep 08 13:44:36.931162 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.930759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" event={"ID":"c26bcd53-9ee2-4082-809b-b28852380d9a","Type":"ContainerDied","Data":"da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7"}
Sep 08 13:44:36.931162 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.931117    2561 scope.go:117] "RemoveContainer" containerID="da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7"
Sep 08 13:44:36.932309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.932274    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:36.932426 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.932347    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:36.932426 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.932394    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:36.942560 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.942535    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"]
Sep 08 13:44:36.950094 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.950072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"
Sep 08 13:44:36.952951 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.952932    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 13:44:36.953032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.953022    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 13:44:36.957731 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:36.957710    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"]
Sep 08 13:44:37.065803 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:37.065768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2617ec-f6e2-4025-8f18-9afa8aad014a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm\" (UID: \"1f2617ec-f6e2-4025-8f18-9afa8aad014a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"
Sep 08 13:44:37.166338 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:37.166285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2617ec-f6e2-4025-8f18-9afa8aad014a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm\" (UID: \"1f2617ec-f6e2-4025-8f18-9afa8aad014a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"
Sep 08 13:44:37.167166 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:37.167140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2617ec-f6e2-4025-8f18-9afa8aad014a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm\" (UID: \"1f2617ec-f6e2-4025-8f18-9afa8aad014a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"
Sep 08 13:44:37.274224 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:37.274143    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"
Sep 08 13:44:37.570092 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:37.570068    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"]
Sep 08 13:44:37.572499 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:37.572476    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2617ec_f6e2_4025_8f18_9afa8aad014a.slice/crio-46f37cefddfbbf7270dec48a81306047c778fbbb32f4fa34754b15fbbb8b7c71 WatchSource:0}: Error finding container 46f37cefddfbbf7270dec48a81306047c778fbbb32f4fa34754b15fbbb8b7c71: Status 404 returned error can't find the container with id 46f37cefddfbbf7270dec48a81306047c778fbbb32f4fa34754b15fbbb8b7c71
Sep 08 13:44:37.938940 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:37.938893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" event={"ID":"c26bcd53-9ee2-4082-809b-b28852380d9a","Type":"ContainerStarted","Data":"67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3"}
Sep 08 13:44:37.941707 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:37.941660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm" event={"ID":"1f2617ec-f6e2-4025-8f18-9afa8aad014a","Type":"ContainerStarted","Data":"46f37cefddfbbf7270dec48a81306047c778fbbb32f4fa34754b15fbbb8b7c71"}
Sep 08 13:44:38.142782 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:38.142745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:38.144163 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:38.144131    2561 patch_prober.go:28] interesting pod/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:44:38.144302 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:38.144194    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 08 13:44:39.153400 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:39.152785    2561 patch_prober.go:28] interesting pod/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:44:39.153400 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:39.152879    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 08 13:44:40.024226 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.024193    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"]
Sep 08 13:44:40.027436 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.027416    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.030094 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.030070    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-c2512ae8-kserve-self-signed-certs\""
Sep 08 13:44:40.062037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.062002    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"]
Sep 08 13:44:40.158405 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.150770    2561 patch_prober.go:28] interesting pod/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:44:40.158405 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.150831    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 08 13:44:40.192045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.192007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-home\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.192226 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.192055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-model-cache\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.192226 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.192118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-dshm\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.192226 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.192197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tmp-dir\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.192356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.192227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbknj\" (UniqueName: \"kubernetes.io/projected/ca4d15fa-1034-4bb1-b58b-a53819eaa407-kube-api-access-fbknj\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.192356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.192275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tls-certs\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.293132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.293040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-home\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.293132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.293076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-model-cache\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.293481 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.293455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-home\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.293575 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.293496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-model-cache\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.293575 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.293524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-dshm\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.293699 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.293614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tmp-dir\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.293699 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.293643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbknj\" (UniqueName: \"kubernetes.io/projected/ca4d15fa-1034-4bb1-b58b-a53819eaa407-kube-api-access-fbknj\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.293699 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.293693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tls-certs\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.294074 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.294003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tmp-dir\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.295749 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.295728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-dshm\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.296434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.296415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tls-certs\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.311776 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.311722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbknj\" (UniqueName: \"kubernetes.io/projected/ca4d15fa-1034-4bb1-b58b-a53819eaa407-kube-api-access-fbknj\") pod \"e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") " pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.366593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.366562    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:40.649007 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.648685    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"]
Sep 08 13:44:40.777251 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.777205    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:40.975424 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.975388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m" event={"ID":"ca4d15fa-1034-4bb1-b58b-a53819eaa407","Type":"ContainerStarted","Data":"c1b3d87e654c6446d4d30bbc38cdafd934e1ed33ff5beedfdaa0f30cd1031ce8"}
Sep 08 13:44:40.975628 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.975433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m" event={"ID":"ca4d15fa-1034-4bb1-b58b-a53819eaa407","Type":"ContainerStarted","Data":"2eaacd2787ece7b8fd755fca6db820af67dbf0d993ff1b6d77a2f6b334842457"}
Sep 08 13:44:40.976959 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.976929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm" event={"ID":"1f2617ec-f6e2-4025-8f18-9afa8aad014a","Type":"ContainerStarted","Data":"91670510832b9d516d1a7b7db99f44d1f692d8b86ddee28d10477d2e5bb01f46"}
Sep 08 13:44:40.993486 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:40.993435    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m" podStartSLOduration=0.993418791 podStartE2EDuration="993.418791ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:40.992869443 +0000 UTC m=+989.343855391" watchObservedRunningTime="2026-09-08 13:44:40.993418791 +0000 UTC m=+989.344404739"
Sep 08 13:44:41.009239 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:41.009192    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm" podStartSLOduration=2.305987065 podStartE2EDuration="5.009177063s" totalImagesPullingTime="2.703189998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:41.008766622 +0000 UTC m=+989.359752582" watchObservedRunningTime="2026-09-08 13:44:41.009177063 +0000 UTC m=+989.360163010"
Sep 08 13:44:41.142604 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:41.142562    2561 patch_prober.go:28] interesting pod/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:44:41.142795 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:41.142630    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 08 13:44:41.983792 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:41.983751    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f2617ec-f6e2-4025-8f18-9afa8aad014a" containerID="91670510832b9d516d1a7b7db99f44d1f692d8b86ddee28d10477d2e5bb01f46" exitCode=0
Sep 08 13:44:41.984239 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:41.983836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm" event={"ID":"1f2617ec-f6e2-4025-8f18-9afa8aad014a","Type":"ContainerDied","Data":"91670510832b9d516d1a7b7db99f44d1f692d8b86ddee28d10477d2e5bb01f46"}
Sep 08 13:44:42.142728 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:42.142692    2561 patch_prober.go:28] interesting pod/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:44:42.143012 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:42.142984    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 08 13:44:43.118375 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.118349    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"
Sep 08 13:44:43.223149 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.223116    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2617ec-f6e2-4025-8f18-9afa8aad014a-maas-api-service-ca\") pod \"1f2617ec-f6e2-4025-8f18-9afa8aad014a\" (UID: \"1f2617ec-f6e2-4025-8f18-9afa8aad014a\") "
Sep 08 13:44:43.223487 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.223464    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2617ec-f6e2-4025-8f18-9afa8aad014a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f2617ec-f6e2-4025-8f18-9afa8aad014a" (UID: "1f2617ec-f6e2-4025-8f18-9afa8aad014a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:44:43.306716 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.306633    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:43.308126 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.308103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:44:43.324495 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.324468    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2617ec-f6e2-4025-8f18-9afa8aad014a-maas-api-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:43.999552 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.999520    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"
Sep 08 13:44:43.999707 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.999556    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm" event={"ID":"1f2617ec-f6e2-4025-8f18-9afa8aad014a","Type":"ContainerDied","Data":"46f37cefddfbbf7270dec48a81306047c778fbbb32f4fa34754b15fbbb8b7c71"}
Sep 08 13:44:43.999707 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:43.999625    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46f37cefddfbbf7270dec48a81306047c778fbbb32f4fa34754b15fbbb8b7c71"
Sep 08 13:44:50.367329 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:50.367288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:50.367780 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:50.367352    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:50.385110 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:50.385031    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:51.043489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:51.043454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:44:53.041556 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.041518    2561 generic.go:356] "Generic (PLEG): container finished" podID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerID="ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5" exitCode=0
Sep 08 13:44:53.041965 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.041590    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" event={"ID":"df004aa6-08ae-47f7-a6b3-c7468e6699f4","Type":"ContainerDied","Data":"ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5"}
Sep 08 13:44:53.041965 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.041946    2561 scope.go:117] "RemoveContainer" containerID="ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5"
Sep 08 13:44:53.042974 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.042949    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:53.043050 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.043013    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:53.043050 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.043044    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:53.422874 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.422848    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"]
Sep 08 13:44:53.423147 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.423110    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6" podUID="d6336460-76de-4d47-9e19-af6ae23ccefb" containerName="payload-processing" containerID="cri-o://6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda" gracePeriod=30
Sep 08 13:44:53.629318 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.629285    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"]
Sep 08 13:44:53.633760 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.633735    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b415db2-ccdsm"]
Sep 08 13:44:53.672693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.672668    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:53.786920 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.786841    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"]
Sep 08 13:44:53.787062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.787046    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy" containerID="cri-o://baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3" gracePeriod=30
Sep 08 13:44:53.826577 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.826548    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88tw8\" (UniqueName: \"kubernetes.io/projected/d6336460-76de-4d47-9e19-af6ae23ccefb-kube-api-access-88tw8\") pod \"d6336460-76de-4d47-9e19-af6ae23ccefb\" (UID: \"d6336460-76de-4d47-9e19-af6ae23ccefb\") "
Sep 08 13:44:53.826704 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.826599    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6336460-76de-4d47-9e19-af6ae23ccefb-plugins-config-volume\") pod \"d6336460-76de-4d47-9e19-af6ae23ccefb\" (UID: \"d6336460-76de-4d47-9e19-af6ae23ccefb\") "
Sep 08 13:44:53.826991 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.826969    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6336460-76de-4d47-9e19-af6ae23ccefb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6336460-76de-4d47-9e19-af6ae23ccefb" (UID: "d6336460-76de-4d47-9e19-af6ae23ccefb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:44:53.828659 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.828634    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6336460-76de-4d47-9e19-af6ae23ccefb-kube-api-access-88tw8" (OuterVolumeSpecName: "kube-api-access-88tw8") pod "d6336460-76de-4d47-9e19-af6ae23ccefb" (UID: "d6336460-76de-4d47-9e19-af6ae23ccefb"). InnerVolumeSpecName "kube-api-access-88tw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:44:53.927713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.927678    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88tw8\" (UniqueName: \"kubernetes.io/projected/d6336460-76de-4d47-9e19-af6ae23ccefb-kube-api-access-88tw8\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:53.927713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:53.927707    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6336460-76de-4d47-9e19-af6ae23ccefb-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:54.051461 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.051343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" event={"ID":"df004aa6-08ae-47f7-a6b3-c7468e6699f4","Type":"ContainerStarted","Data":"bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b"}
Sep 08 13:44:54.053817 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.053782    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6336460-76de-4d47-9e19-af6ae23ccefb" containerID="6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda" exitCode=0
Sep 08 13:44:54.053948 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.053828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6" event={"ID":"d6336460-76de-4d47-9e19-af6ae23ccefb","Type":"ContainerDied","Data":"6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda"}
Sep 08 13:44:54.053948 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.053871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6" event={"ID":"d6336460-76de-4d47-9e19-af6ae23ccefb","Type":"ContainerDied","Data":"f55bb167e698cd5c45216d3e84c1d14a1ef01503e6a91282e4efd7052c3d3fac"}
Sep 08 13:44:54.053948 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.053902    2561 scope.go:117] "RemoveContainer" containerID="6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda"
Sep 08 13:44:54.054111 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.054056    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"
Sep 08 13:44:54.073299 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.073278    2561 scope.go:117] "RemoveContainer" containerID="6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda"
Sep 08 13:44:54.073624 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:44:54.073602    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda\": container with ID starting with 6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda not found: ID does not exist" containerID="6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda"
Sep 08 13:44:54.073702 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.073633    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda"} err="failed to get container status \"6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda\": rpc error: code = NotFound desc = could not find container \"6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda\": container with ID starting with 6e5ede37fea76c8a93d23afd30236e243c01473970c49ba2a4ea2b9c7d6e1cda not found: ID does not exist"
Sep 08 13:44:54.096287 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.096262    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"]
Sep 08 13:44:54.100489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.100470    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0b415db2-5cd8d6cf55-kkkk6"]
Sep 08 13:44:54.248843 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.248799    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f2617ec-f6e2-4025-8f18-9afa8aad014a" path="/var/lib/kubelet/pods/1f2617ec-f6e2-4025-8f18-9afa8aad014a/volumes"
Sep 08 13:44:54.249369 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.249341    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6336460-76de-4d47-9e19-af6ae23ccefb" path="/var/lib/kubelet/pods/d6336460-76de-4d47-9e19-af6ae23ccefb/volumes"
Sep 08 13:44:54.501551 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.501501    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:44:54.501729 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.501688    2561 patch_prober.go:28] interesting pod/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 08 13:44:54.501767 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.501730    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 08 13:44:54.791701 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.791626    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"]
Sep 08 13:44:54.791960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.791944    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f2617ec-f6e2-4025-8f18-9afa8aad014a" containerName="revoke-keys"
Sep 08 13:44:54.792001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.791963    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f2617ec-f6e2-4025-8f18-9afa8aad014a" containerName="revoke-keys"
Sep 08 13:44:54.792001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.791980    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6336460-76de-4d47-9e19-af6ae23ccefb" containerName="payload-processing"
Sep 08 13:44:54.792001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.791986    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6336460-76de-4d47-9e19-af6ae23ccefb" containerName="payload-processing"
Sep 08 13:44:54.792092 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.792037    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f2617ec-f6e2-4025-8f18-9afa8aad014a" containerName="revoke-keys"
Sep 08 13:44:54.792092 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.792045    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6336460-76de-4d47-9e19-af6ae23ccefb" containerName="payload-processing"
Sep 08 13:44:54.796221 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.796200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.800096 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.800077    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fed6eaa3-gw-openshift-default-dockercfg-htnc2\""
Sep 08 13:44:54.812866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.812839    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"]
Sep 08 13:44:54.855403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-credential-socket\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.855403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0db05-16c1-42e1-8284-7857a2894753-istio-podinfo\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.855403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-certs\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.855403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-data\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.855403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0db05-16c1-42e1-8284-7857a2894753-istiod-ca-cert\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.855825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9nw5\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-kube-api-access-g9nw5\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.855825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-envoy\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.855825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-istio-token\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.855825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.855554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-socket\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.956674 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.956634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-credential-socket\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.957006 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.956978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0db05-16c1-42e1-8284-7857a2894753-istio-podinfo\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.957167 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-certs\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.957298 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-data\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.957455 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0db05-16c1-42e1-8284-7857a2894753-istiod-ca-cert\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.957583 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9nw5\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-kube-api-access-g9nw5\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.957704 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-envoy\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.957815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-istio-token\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.957944 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-socket\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.958230 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-certs\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.958325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-credential-socket\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.958325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.957731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-data\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.958470 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.958337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0db05-16c1-42e1-8284-7857a2894753-istiod-ca-cert\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.958990 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.958969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-socket\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.960317 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.960290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0db05-16c1-42e1-8284-7857a2894753-istio-podinfo\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.961737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.961702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-envoy\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.976159 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.976132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9nw5\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-kube-api-access-g9nw5\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:54.976473 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:54.976452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-istio-token\") pod \"e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") " pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:55.106257 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:55.106178    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:55.404493 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:55.404468    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"]
Sep 08 13:44:55.406122 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:55.406099    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce0db05_16c1_42e1_8284_7857a2894753.slice/crio-0b8519169e44a5437abd342ca13090972add1a29bb4c431b6294cfd98bf9532f WatchSource:0}: Error finding container 0b8519169e44a5437abd342ca13090972add1a29bb4c431b6294cfd98bf9532f: Status 404 returned error can't find the container with id 0b8519169e44a5437abd342ca13090972add1a29bb4c431b6294cfd98bf9532f
Sep 08 13:44:55.408424 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:55.408371    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:55.408519 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:55.408468    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:55.408519 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:55.408513    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:44:55.501677 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:55.501648    2561 patch_prober.go:28] interesting pod/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 08 13:44:55.501810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:55.501694    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 08 13:44:56.069447 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:56.069408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" event={"ID":"4ce0db05-16c1-42e1-8284-7857a2894753","Type":"ContainerStarted","Data":"a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4"}
Sep 08 13:44:56.069652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:56.069468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" event={"ID":"4ce0db05-16c1-42e1-8284-7857a2894753","Type":"ContainerStarted","Data":"0b8519169e44a5437abd342ca13090972add1a29bb4c431b6294cfd98bf9532f"}
Sep 08 13:44:56.096063 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:56.096004    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podStartSLOduration=2.095986234 podStartE2EDuration="2.095986234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:56.094707545 +0000 UTC m=+1004.445693494" watchObservedRunningTime="2026-09-08 13:44:56.095986234 +0000 UTC m=+1004.446972183"
Sep 08 13:44:56.106527 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:56.106497    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:56.108009 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:56.107985    2561 patch_prober.go:28] interesting pod/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 08 13:44:56.108193 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:56.108036    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 08 13:44:56.501600 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:56.501566    2561 patch_prober.go:28] interesting pod/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 08 13:44:56.501771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:56.501620    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 08 13:44:57.110437 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:57.110403    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:57.501737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:57.501700    2561 patch_prober.go:28] interesting pod/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 08 13:44:57.501945 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:57.501776    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 08 13:44:58.080669 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.080638    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:58.082122 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.082098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:44:58.469585 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.469548    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"]
Sep 08 13:44:58.472702 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.472687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:44:58.476650 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.476625    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fed6eaa3-dockercfg-lgqdx\""
Sep 08 13:44:58.476762 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.476711    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fed6eaa3\""
Sep 08 13:44:58.485495 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.485473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf\" (UID: \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:44:58.485593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.485528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6djqn\" (UniqueName: \"kubernetes.io/projected/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-kube-api-access-6djqn\") pod \"payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf\" (UID: \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:44:58.491282 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.491257    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"]
Sep 08 13:44:58.501090 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.501066    2561 patch_prober.go:28] interesting pod/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 08 13:44:58.501187 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.501121    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 08 13:44:58.503877 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.503852    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"]
Sep 08 13:44:58.507722 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.507647    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:44:58.520144 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.520119    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"]
Sep 08 13:44:58.586090 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.586057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf\" (UID: \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:44:58.586278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.586157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6\" (UID: \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:44:58.586278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.586187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6djqn\" (UniqueName: \"kubernetes.io/projected/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-kube-api-access-6djqn\") pod \"payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf\" (UID: \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:44:58.586278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.586214    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx2qp\" (UniqueName: \"kubernetes.io/projected/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-kube-api-access-tx2qp\") pod \"payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6\" (UID: \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:44:58.587006 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.586982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf\" (UID: \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:44:58.602526 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.602493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6djqn\" (UniqueName: \"kubernetes.io/projected/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-kube-api-access-6djqn\") pod \"payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf\" (UID: \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:44:58.687089 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.687050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6\" (UID: \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:44:58.687089 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.687095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx2qp\" (UniqueName: \"kubernetes.io/projected/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-kube-api-access-tx2qp\") pod \"payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6\" (UID: \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:44:58.687791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.687771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6\" (UID: \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:44:58.707360 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.707338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx2qp\" (UniqueName: \"kubernetes.io/projected/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-kube-api-access-tx2qp\") pod \"payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6\" (UID: \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:44:58.783037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.782952    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:44:58.821357 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:58.821326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:44:59.070195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.070172    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:59.074803 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.074709    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"]
Sep 08 13:44:59.076869 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:59.076837    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1cd6ed_85d0_478b_a0be_8b2ef45464a2.slice/crio-7cd021b0bc9f314a54328a05f9031ecf070703ae8bfab4dc338bb5df370cf1b3 WatchSource:0}: Error finding container 7cd021b0bc9f314a54328a05f9031ecf070703ae8bfab4dc338bb5df370cf1b3: Status 404 returned error can't find the container with id 7cd021b0bc9f314a54328a05f9031ecf070703ae8bfab4dc338bb5df370cf1b3
Sep 08 13:44:59.085820 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.085790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf" event={"ID":"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2","Type":"ContainerStarted","Data":"7cd021b0bc9f314a54328a05f9031ecf070703ae8bfab4dc338bb5df370cf1b3"}
Sep 08 13:44:59.087500 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.087474    2561 generic.go:356] "Generic (PLEG): container finished" podID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerID="baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3" exitCode=0
Sep 08 13:44:59.087607 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.087524    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"
Sep 08 13:44:59.087607 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.087547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" event={"ID":"86a0eee6-0d1f-41f7-9f13-6de3b476be44","Type":"ContainerDied","Data":"baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3"}
Sep 08 13:44:59.087607 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.087586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g" event={"ID":"86a0eee6-0d1f-41f7-9f13-6de3b476be44","Type":"ContainerDied","Data":"9ab4031e0660b572e515df16c0c72f795537be5b7db17e534539f934f4250209"}
Sep 08 13:44:59.087763 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.087612    2561 scope.go:117] "RemoveContainer" containerID="baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3"
Sep 08 13:44:59.091178 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.091160    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-envoy\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.091263 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.091193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-certs\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.091684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.091561    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:44:59.093997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.093952    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:44:59.099811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.099790    2561 scope.go:117] "RemoveContainer" containerID="4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18"
Sep 08 13:44:59.113122 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.113095    2561 scope.go:117] "RemoveContainer" containerID="baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3"
Sep 08 13:44:59.113474 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:44:59.113451    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3\": container with ID starting with baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3 not found: ID does not exist" containerID="baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3"
Sep 08 13:44:59.113568 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.113481    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3"} err="failed to get container status \"baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3\": rpc error: code = NotFound desc = could not find container \"baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3\": container with ID starting with baf0edb257cb05705413e3bf314f091e91fd7e4928d3daa2d5ff54be84583dd3 not found: ID does not exist"
Sep 08 13:44:59.113568 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.113499    2561 scope.go:117] "RemoveContainer" containerID="4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18"
Sep 08 13:44:59.113836 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:44:59.113808    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18\": container with ID starting with 4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18 not found: ID does not exist" containerID="4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18"
Sep 08 13:44:59.113933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.113846    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18"} err="failed to get container status \"4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18\": rpc error: code = NotFound desc = could not find container \"4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18\": container with ID starting with 4b7db06d20928e9b1c219dffd531bd29dbe3427cea55a5c8ca52828c2b633b18 not found: ID does not exist"
Sep 08 13:44:59.150000 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.136121    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"]
Sep 08 13:44:59.162667 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:44:59.162590    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba609fc7_b0b1_46e8_b0c0_0568dc9ddcb5.slice/crio-c37bf66b8228565e899890bd5e02e59a42c14cef6bf038d88ab71fae2a27f996 WatchSource:0}: Error finding container c37bf66b8228565e899890bd5e02e59a42c14cef6bf038d88ab71fae2a27f996: Status 404 returned error can't find the container with id c37bf66b8228565e899890bd5e02e59a42c14cef6bf038d88ab71fae2a27f996
Sep 08 13:44:59.191894 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.191861    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istiod-ca-cert\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.192008 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.191945    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-podinfo\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.192008 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.191980    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-credential-socket\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.192008 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192004    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-socket\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.192109 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-token\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.192109 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqz8w\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-kube-api-access-cqz8w\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.192109 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192081    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-data\") pod \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\" (UID: \"86a0eee6-0d1f-41f7-9f13-6de3b476be44\") "
Sep 08 13:44:59.192222 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:44:59.192603 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-data" (OuterVolumeSpecName: "istio-data") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:44:59.192603 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192450    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.192603 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192475    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.192603 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192488    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.192603 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:44:59.192923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.192634    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:44:59.194612 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.194589    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-token" (OuterVolumeSpecName: "istio-token") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:44:59.194925 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.194901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-kube-api-access-cqz8w" (OuterVolumeSpecName: "kube-api-access-cqz8w") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "kube-api-access-cqz8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:44:59.195132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.195112    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86a0eee6-0d1f-41f7-9f13-6de3b476be44" (UID: "86a0eee6-0d1f-41f7-9f13-6de3b476be44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:44:59.293764 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.293729    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.293764 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.293765    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.293950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.293783    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.293950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.293797    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.293950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.293811    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqz8w\" (UniqueName: \"kubernetes.io/projected/86a0eee6-0d1f-41f7-9f13-6de3b476be44-kube-api-access-cqz8w\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.293950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.293824    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86a0eee6-0d1f-41f7-9f13-6de3b476be44-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:44:59.413653 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.413622    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"]
Sep 08 13:44:59.420194 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.420165    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0b415db2-openshift-default-779f5dcffb-q6q2g"]
Sep 08 13:44:59.502256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.502214    2561 patch_prober.go:28] interesting pod/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 08 13:44:59.502750 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:44:59.502280    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 08 13:45:00.094724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.094682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6" event={"ID":"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5","Type":"ContainerStarted","Data":"0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a"}
Sep 08 13:45:00.094917 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.094741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6" event={"ID":"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5","Type":"ContainerStarted","Data":"c37bf66b8228565e899890bd5e02e59a42c14cef6bf038d88ab71fae2a27f996"}
Sep 08 13:45:00.094917 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.094888    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:45:00.096202 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.096177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf" event={"ID":"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2","Type":"ContainerStarted","Data":"2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641"}
Sep 08 13:45:00.096320 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.096214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:45:00.113903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.113863    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6" podStartSLOduration=2.113851895 podStartE2EDuration="2.113851895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:00.11222079 +0000 UTC m=+1008.463206734" watchObservedRunningTime="2026-09-08 13:45:00.113851895 +0000 UTC m=+1008.464837876"
Sep 08 13:45:00.139998 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.139951    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf" podStartSLOduration=2.139937442 podStartE2EDuration="2.139937442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:00.137826556 +0000 UTC m=+1008.488812503" watchObservedRunningTime="2026-09-08 13:45:00.139937442 +0000 UTC m=+1008.490923390"
Sep 08 13:45:00.142863 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.142838    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"]
Sep 08 13:45:00.143274 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.143258    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy"
Sep 08 13:45:00.143366 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.143277    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy"
Sep 08 13:45:00.143419 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.143370    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy"
Sep 08 13:45:00.143419 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.143396    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy"
Sep 08 13:45:00.146511 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.146491    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"]
Sep 08 13:45:00.146621 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.146605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"
Sep 08 13:45:00.146905 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.146892    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy"
Sep 08 13:45:00.146953 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.146907    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" containerName="istio-proxy"
Sep 08 13:45:00.149184 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.149165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 13:45:00.149802 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.149786    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"]
Sep 08 13:45:00.149896 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.149884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"
Sep 08 13:45:00.153092 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.153073    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"
Sep 08 13:45:00.157640 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.157613    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"]
Sep 08 13:45:00.158669 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.158527    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"
Sep 08 13:45:00.159846 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.159825    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"]
Sep 08 13:45:00.169454 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.164147    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"]
Sep 08 13:45:00.174419 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.174371    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"
Sep 08 13:45:00.208878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.208847    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"
Sep 08 13:45:00.250838 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.250757    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86a0eee6-0d1f-41f7-9f13-6de3b476be44" path="/var/lib/kubelet/pods/86a0eee6-0d1f-41f7-9f13-6de3b476be44/volumes"
Sep 08 13:45:00.447174 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.447131    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"]
Sep 08 13:45:00.452229 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:00.452199    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35380b5_f1e2_4e94_b168_07390ac41a88.slice/crio-4c0ed0ab57993630cfee2fe8b5cebc4758f75966bd61852f9dedb5b190accd12 WatchSource:0}: Error finding container 4c0ed0ab57993630cfee2fe8b5cebc4758f75966bd61852f9dedb5b190accd12: Status 404 returned error can't find the container with id 4c0ed0ab57993630cfee2fe8b5cebc4758f75966bd61852f9dedb5b190accd12
Sep 08 13:45:00.456568 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.456543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"]
Sep 08 13:45:00.458125 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:00.458100    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907a5911_8ad8_4a6f_a2a3_9e4d7ad27a7b.slice/crio-7438b83c3af5abc73f6022c7f376233afd5010a16475f43c30951a1fa5b14765 WatchSource:0}: Error finding container 7438b83c3af5abc73f6022c7f376233afd5010a16475f43c30951a1fa5b14765: Status 404 returned error can't find the container with id 7438b83c3af5abc73f6022c7f376233afd5010a16475f43c30951a1fa5b14765
Sep 08 13:45:00.503300 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.503244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"]
Sep 08 13:45:00.505327 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:00.505303    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250f58a8_dbcb_415b_ad28_bd1c5d653597.slice/crio-c7f47240468f9246867328454016a025d40df1a2b2400f49b6892d738c473802 WatchSource:0}: Error finding container c7f47240468f9246867328454016a025d40df1a2b2400f49b6892d738c473802: Status 404 returned error can't find the container with id c7f47240468f9246867328454016a025d40df1a2b2400f49b6892d738c473802
Sep 08 13:45:00.506457 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.506437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:45:00.506741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.506723    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:45:00.507601 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:00.507582    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:45:01.102522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.102429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26" event={"ID":"907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b","Type":"ContainerStarted","Data":"61c491a0494023e87010f85511e3dd649b1b6156c33b6677fbfef760cd82abe1"}
Sep 08 13:45:01.102522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.102479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26" event={"ID":"907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b","Type":"ContainerStarted","Data":"7438b83c3af5abc73f6022c7f376233afd5010a16475f43c30951a1fa5b14765"}
Sep 08 13:45:01.103974 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.103946    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" event={"ID":"250f58a8-dbcb-415b-ad28-bd1c5d653597","Type":"ContainerStarted","Data":"b981019b4240b6292194cc034c9aadff0204eebd6f658476f259355ab4412aff"}
Sep 08 13:45:01.104100 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.103997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" event={"ID":"250f58a8-dbcb-415b-ad28-bd1c5d653597","Type":"ContainerStarted","Data":"c7f47240468f9246867328454016a025d40df1a2b2400f49b6892d738c473802"}
Sep 08 13:45:01.105432 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.105402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6" event={"ID":"e35380b5-f1e2-4e94-b168-07390ac41a88","Type":"ContainerStarted","Data":"e38f9adfd0ed38c47fb59f71a0550835afa6244da6ed7bda7f5e8ae4ce1f2a0e"}
Sep 08 13:45:01.105536 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.105454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6" event={"ID":"e35380b5-f1e2-4e94-b168-07390ac41a88","Type":"ContainerStarted","Data":"4c0ed0ab57993630cfee2fe8b5cebc4758f75966bd61852f9dedb5b190accd12"}
Sep 08 13:45:01.119623 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.119577    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26" podStartSLOduration=1.119563192 podStartE2EDuration="1.119563192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:01.118741595 +0000 UTC m=+1009.469727543" watchObservedRunningTime="2026-09-08 13:45:01.119563192 +0000 UTC m=+1009.470549140"
Sep 08 13:45:01.135734 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.135683    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6" podStartSLOduration=1.135668712 podStartE2EDuration="1.135668712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:01.134290106 +0000 UTC m=+1009.485276053" watchObservedRunningTime="2026-09-08 13:45:01.135668712 +0000 UTC m=+1009.486654659"
Sep 08 13:45:01.153760 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:01.153708    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" podStartSLOduration=1.153691821 podStartE2EDuration="1.153691821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:01.151904472 +0000 UTC m=+1009.502890429" watchObservedRunningTime="2026-09-08 13:45:01.153691821 +0000 UTC m=+1009.504677769"
Sep 08 13:45:02.111442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:02.111413    2561 generic.go:356] "Generic (PLEG): container finished" podID="907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b" containerID="61c491a0494023e87010f85511e3dd649b1b6156c33b6677fbfef760cd82abe1" exitCode=0
Sep 08 13:45:02.111848 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:02.111497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26" event={"ID":"907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b","Type":"ContainerDied","Data":"61c491a0494023e87010f85511e3dd649b1b6156c33b6677fbfef760cd82abe1"}
Sep 08 13:45:02.112966 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:02.112943    2561 generic.go:356] "Generic (PLEG): container finished" podID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerID="b981019b4240b6292194cc034c9aadff0204eebd6f658476f259355ab4412aff" exitCode=7
Sep 08 13:45:02.113077 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:02.113034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" event={"ID":"250f58a8-dbcb-415b-ad28-bd1c5d653597","Type":"ContainerDied","Data":"b981019b4240b6292194cc034c9aadff0204eebd6f658476f259355ab4412aff"}
Sep 08 13:45:02.113155 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:02.113138    2561 scope.go:117] "RemoveContainer" containerID="b981019b4240b6292194cc034c9aadff0204eebd6f658476f259355ab4412aff"
Sep 08 13:45:02.114575 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:02.114554    2561 generic.go:356] "Generic (PLEG): container finished" podID="e35380b5-f1e2-4e94-b168-07390ac41a88" containerID="e38f9adfd0ed38c47fb59f71a0550835afa6244da6ed7bda7f5e8ae4ce1f2a0e" exitCode=0
Sep 08 13:45:02.114652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:02.114604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6" event={"ID":"e35380b5-f1e2-4e94-b168-07390ac41a88","Type":"ContainerDied","Data":"e38f9adfd0ed38c47fb59f71a0550835afa6244da6ed7bda7f5e8ae4ce1f2a0e"}
Sep 08 13:45:03.121691 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:03.121607    2561 generic.go:356] "Generic (PLEG): container finished" podID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerID="65de28efc4857f5718936968c06451c5487d0950245bd91746a534a6076d426e" exitCode=7
Sep 08 13:45:03.122135 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:03.121691    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" event={"ID":"250f58a8-dbcb-415b-ad28-bd1c5d653597","Type":"ContainerDied","Data":"65de28efc4857f5718936968c06451c5487d0950245bd91746a534a6076d426e"}
Sep 08 13:45:03.122135 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:03.121749    2561 scope.go:117] "RemoveContainer" containerID="b981019b4240b6292194cc034c9aadff0204eebd6f658476f259355ab4412aff"
Sep 08 13:45:03.122135 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:03.121901    2561 scope.go:117] "RemoveContainer" containerID="65de28efc4857f5718936968c06451c5487d0950245bd91746a534a6076d426e"
Sep 08 13:45:03.122285 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:45:03.122153    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7_odh-ai-gateway-infra(250f58a8-dbcb-415b-ad28-bd1c5d653597)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597"
Sep 08 13:45:03.264811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:03.264784    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"
Sep 08 13:45:03.287775 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:03.287750    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"
Sep 08 13:45:04.127106 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:04.127071    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"
Sep 08 13:45:04.127584 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:04.127122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26" event={"ID":"907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b","Type":"ContainerDied","Data":"7438b83c3af5abc73f6022c7f376233afd5010a16475f43c30951a1fa5b14765"}
Sep 08 13:45:04.127584 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:04.127205    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7438b83c3af5abc73f6022c7f376233afd5010a16475f43c30951a1fa5b14765"
Sep 08 13:45:04.128884 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:04.128864    2561 scope.go:117] "RemoveContainer" containerID="65de28efc4857f5718936968c06451c5487d0950245bd91746a534a6076d426e"
Sep 08 13:45:04.129102 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:45:04.129076    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7_odh-ai-gateway-infra(250f58a8-dbcb-415b-ad28-bd1c5d653597)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597"
Sep 08 13:45:04.129889 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:04.129869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6" event={"ID":"e35380b5-f1e2-4e94-b168-07390ac41a88","Type":"ContainerDied","Data":"4c0ed0ab57993630cfee2fe8b5cebc4758f75966bd61852f9dedb5b190accd12"}
Sep 08 13:45:04.129889 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:04.129886    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"
Sep 08 13:45:04.130005 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:04.129910    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c0ed0ab57993630cfee2fe8b5cebc4758f75966bd61852f9dedb5b190accd12"
Sep 08 13:45:11.107294 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:11.107261    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:45:11.107762 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:11.107350    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:45:16.247832 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:16.247804    2561 scope.go:117] "RemoveContainer" containerID="65de28efc4857f5718936968c06451c5487d0950245bd91746a534a6076d426e"
Sep 08 13:45:17.202915 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:17.202877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" event={"ID":"250f58a8-dbcb-415b-ad28-bd1c5d653597","Type":"ContainerStarted","Data":"ab34a5f0c973415a35bbe24df9253fb53b5b9261c909d8b3ed79a2b378e401ef"}
Sep 08 13:45:18.210486 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:18.210454    2561 generic.go:356] "Generic (PLEG): container finished" podID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerID="ab34a5f0c973415a35bbe24df9253fb53b5b9261c909d8b3ed79a2b378e401ef" exitCode=0
Sep 08 13:45:18.210946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:18.210528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" event={"ID":"250f58a8-dbcb-415b-ad28-bd1c5d653597","Type":"ContainerDied","Data":"ab34a5f0c973415a35bbe24df9253fb53b5b9261c909d8b3ed79a2b378e401ef"}
Sep 08 13:45:18.210946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:18.210608    2561 scope.go:117] "RemoveContainer" containerID="65de28efc4857f5718936968c06451c5487d0950245bd91746a534a6076d426e"
Sep 08 13:45:18.236937 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:18.236906    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"]
Sep 08 13:45:19.216690 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:19.216655    2561 generic.go:356] "Generic (PLEG): container finished" podID="4ce0db05-16c1-42e1-8284-7857a2894753" containerID="a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4" exitCode=0
Sep 08 13:45:19.217234 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:19.216731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" event={"ID":"4ce0db05-16c1-42e1-8284-7857a2894753","Type":"ContainerDied","Data":"a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4"}
Sep 08 13:45:19.217307 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:19.217236    2561 scope.go:117] "RemoveContainer" containerID="a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4"
Sep 08 13:45:19.218427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:19.218398    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:19.218540 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:19.218466    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:19.218540 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:19.218497    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:19.349090 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:19.349062    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"
Sep 08 13:45:20.224736 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:20.224702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" event={"ID":"4ce0db05-16c1-42e1-8284-7857a2894753","Type":"ContainerStarted","Data":"c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b"}
Sep 08 13:45:20.226004 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:20.225985    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"
Sep 08 13:45:20.226107 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:20.226002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7" event={"ID":"250f58a8-dbcb-415b-ad28-bd1c5d653597","Type":"ContainerDied","Data":"c7f47240468f9246867328454016a025d40df1a2b2400f49b6892d738c473802"}
Sep 08 13:45:20.226107 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:20.226039    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7f47240468f9246867328454016a025d40df1a2b2400f49b6892d738c473802"
Sep 08 13:45:20.278008 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:20.277976    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"]
Sep 08 13:45:20.284162 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:20.284137    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-fed6eaa3-29814585-b2vh7"]
Sep 08 13:45:21.106367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:21.106333    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:45:21.106534 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:21.106495    2561 patch_prober.go:28] interesting pod/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 08 13:45:21.106593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:21.106548    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 08 13:45:22.107004 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.106967    2561 patch_prober.go:28] interesting pod/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 08 13:45:22.107492 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.107022    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 08 13:45:22.247728 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.247693    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" path="/var/lib/kubelet/pods/250f58a8-dbcb-415b-ad28-bd1c5d653597/volumes"
Sep 08 13:45:22.606694 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.606656    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"]
Sep 08 13:45:22.606983 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.606971    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b" containerName="cleanup"
Sep 08 13:45:22.607032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.606984    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b" containerName="cleanup"
Sep 08 13:45:22.607032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.606994    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:22.607032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607000    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:22.607032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607008    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:22.607032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607015    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:22.607032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607025    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35380b5-f1e2-4e94-b168-07390ac41a88" containerName="cleanup"
Sep 08 13:45:22.607032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607030    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35380b5-f1e2-4e94-b168-07390ac41a88" containerName="cleanup"
Sep 08 13:45:22.607234 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607081    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35380b5-f1e2-4e94-b168-07390ac41a88" containerName="cleanup"
Sep 08 13:45:22.607234 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607087    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:22.607234 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607094    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b" containerName="cleanup"
Sep 08 13:45:22.607234 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.607103    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:22.610207 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.610189    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"
Sep 08 13:45:22.612776 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.612756    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 13:45:22.612895 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.612792    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 13:45:22.618454 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.618436    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"]
Sep 08 13:45:22.705205 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.705168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e801da-a02b-441b-b316-5729c4f4ee16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d\" (UID: \"c2e801da-a02b-441b-b316-5729c4f4ee16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"
Sep 08 13:45:22.806021 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.805983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e801da-a02b-441b-b316-5729c4f4ee16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d\" (UID: \"c2e801da-a02b-441b-b316-5729c4f4ee16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"
Sep 08 13:45:22.806701 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.806683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e801da-a02b-441b-b316-5729c4f4ee16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d\" (UID: \"c2e801da-a02b-441b-b316-5729c4f4ee16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"
Sep 08 13:45:22.920915 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:22.920886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"
Sep 08 13:45:23.107155 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:23.107119    2561 patch_prober.go:28] interesting pod/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 08 13:45:23.107576 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:23.107172    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 08 13:45:23.212882 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:23.212848    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"]
Sep 08 13:45:23.214424 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:23.214392    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e801da_a02b_441b_b316_5729c4f4ee16.slice/crio-86a71d9f1d7ffd4dd2d2a15370f5167ef3ef15dd8336114928154521ee84052d WatchSource:0}: Error finding container 86a71d9f1d7ffd4dd2d2a15370f5167ef3ef15dd8336114928154521ee84052d: Status 404 returned error can't find the container with id 86a71d9f1d7ffd4dd2d2a15370f5167ef3ef15dd8336114928154521ee84052d
Sep 08 13:45:23.260717 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:23.260676    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d" event={"ID":"c2e801da-a02b-441b-b316-5729c4f4ee16","Type":"ContainerStarted","Data":"86a71d9f1d7ffd4dd2d2a15370f5167ef3ef15dd8336114928154521ee84052d"}
Sep 08 13:45:24.107162 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:24.107127    2561 patch_prober.go:28] interesting pod/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 08 13:45:24.107537 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:24.107179    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 08 13:45:24.269559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:24.269526    2561 generic.go:356] "Generic (PLEG): container finished" podID="c2e801da-a02b-441b-b316-5729c4f4ee16" containerID="e4f94232d9676c91bbf0b5ab936bb687c7afc94d5c7ecf330723520c1b9001ab" exitCode=0
Sep 08 13:45:24.269736 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:24.269611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d" event={"ID":"c2e801da-a02b-441b-b316-5729c4f4ee16","Type":"ContainerDied","Data":"e4f94232d9676c91bbf0b5ab936bb687c7afc94d5c7ecf330723520c1b9001ab"}
Sep 08 13:45:25.107439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:25.107402    2561 patch_prober.go:28] interesting pod/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 08 13:45:25.107832 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:25.107461    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 08 13:45:25.412004 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:25.411978    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"
Sep 08 13:45:25.529820 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:25.529784    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e801da-a02b-441b-b316-5729c4f4ee16-maas-api-service-ca\") pod \"c2e801da-a02b-441b-b316-5729c4f4ee16\" (UID: \"c2e801da-a02b-441b-b316-5729c4f4ee16\") "
Sep 08 13:45:25.530104 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:25.530081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e801da-a02b-441b-b316-5729c4f4ee16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2e801da-a02b-441b-b316-5729c4f4ee16" (UID: "c2e801da-a02b-441b-b316-5729c4f4ee16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:25.631024 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:25.630979    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e801da-a02b-441b-b316-5729c4f4ee16-maas-api-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:26.107046 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:26.107008    2561 patch_prober.go:28] interesting pod/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 08 13:45:26.107224 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:26.107066    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 08 13:45:26.280100 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:26.280065    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d" event={"ID":"c2e801da-a02b-441b-b316-5729c4f4ee16","Type":"ContainerDied","Data":"86a71d9f1d7ffd4dd2d2a15370f5167ef3ef15dd8336114928154521ee84052d"}
Sep 08 13:45:26.280480 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:26.280114    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86a71d9f1d7ffd4dd2d2a15370f5167ef3ef15dd8336114928154521ee84052d"
Sep 08 13:45:26.280480 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:26.280117    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"
Sep 08 13:45:27.111564 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:27.111528    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:45:27.111849 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:27.111828    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:45:27.112404 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:27.112363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:45:29.313705 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.313675    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"]
Sep 08 13:45:29.314196 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.313937    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6" podUID="ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5" containerName="payload-processing" containerID="cri-o://0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a" gracePeriod=30
Sep 08 13:45:29.323048 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.322984    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"]
Sep 08 13:45:29.323243 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.323214    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf" podUID="1e1cd6ed-85d0-478b-a0be-8b2ef45464a2" containerName="payload-pre-processing" containerID="cri-o://2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641" gracePeriod=30
Sep 08 13:45:29.549597 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.549567    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"]
Sep 08 13:45:29.551428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.551402    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fed6eaa3-48p5d"]
Sep 08 13:45:29.596223 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.596202    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:45:29.598822 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.598804    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:45:29.669435 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.669409    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx2qp\" (UniqueName: \"kubernetes.io/projected/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-kube-api-access-tx2qp\") pod \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\" (UID: \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\") "
Sep 08 13:45:29.669605 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.669474    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-plugins-config-volume\") pod \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\" (UID: \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\") "
Sep 08 13:45:29.669605 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.669497    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6djqn\" (UniqueName: \"kubernetes.io/projected/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-kube-api-access-6djqn\") pod \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\" (UID: \"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2\") "
Sep 08 13:45:29.669605 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.669592    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-plugins-config-volume\") pod \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\" (UID: \"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5\") "
Sep 08 13:45:29.669904 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.669871    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e1cd6ed-85d0-478b-a0be-8b2ef45464a2" (UID: "1e1cd6ed-85d0-478b-a0be-8b2ef45464a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:29.670009 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.669925    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5" (UID: "ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:29.671565 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.671542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-kube-api-access-6djqn" (OuterVolumeSpecName: "kube-api-access-6djqn") pod "1e1cd6ed-85d0-478b-a0be-8b2ef45464a2" (UID: "1e1cd6ed-85d0-478b-a0be-8b2ef45464a2"). InnerVolumeSpecName "kube-api-access-6djqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:29.671565 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.671549    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-kube-api-access-tx2qp" (OuterVolumeSpecName: "kube-api-access-tx2qp") pod "ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5" (UID: "ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5"). InnerVolumeSpecName "kube-api-access-tx2qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:29.760622 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.758232    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"]
Sep 08 13:45:29.760622 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.758564    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy" containerID="cri-o://c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b" gracePeriod=30
Sep 08 13:45:29.771015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.770949    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx2qp\" (UniqueName: \"kubernetes.io/projected/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-kube-api-access-tx2qp\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:29.771015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.770975    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:29.771015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.770988    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6djqn\" (UniqueName: \"kubernetes.io/projected/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2-kube-api-access-6djqn\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:29.771015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:29.771001    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:30.245613 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.245580    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e801da-a02b-441b-b316-5729c4f4ee16" path="/var/lib/kubelet/pods/c2e801da-a02b-441b-b316-5729c4f4ee16/volumes"
Sep 08 13:45:30.301267 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.301235    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5" containerID="0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a" exitCode=0
Sep 08 13:45:30.301434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.301289    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"
Sep 08 13:45:30.301434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.301317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6" event={"ID":"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5","Type":"ContainerDied","Data":"0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a"}
Sep 08 13:45:30.301434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.301369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6" event={"ID":"ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5","Type":"ContainerDied","Data":"c37bf66b8228565e899890bd5e02e59a42c14cef6bf038d88ab71fae2a27f996"}
Sep 08 13:45:30.301434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.301412    2561 scope.go:117] "RemoveContainer" containerID="0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a"
Sep 08 13:45:30.302566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.302547    2561 generic.go:356] "Generic (PLEG): container finished" podID="1e1cd6ed-85d0-478b-a0be-8b2ef45464a2" containerID="2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641" exitCode=0
Sep 08 13:45:30.302642 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.302612    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"
Sep 08 13:45:30.302732 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.302608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf" event={"ID":"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2","Type":"ContainerDied","Data":"2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641"}
Sep 08 13:45:30.302822 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.302759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf" event={"ID":"1e1cd6ed-85d0-478b-a0be-8b2ef45464a2","Type":"ContainerDied","Data":"7cd021b0bc9f314a54328a05f9031ecf070703ae8bfab4dc338bb5df370cf1b3"}
Sep 08 13:45:30.309867 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.309850    2561 scope.go:117] "RemoveContainer" containerID="0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a"
Sep 08 13:45:30.310115 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:45:30.310097    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a\": container with ID starting with 0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a not found: ID does not exist" containerID="0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a"
Sep 08 13:45:30.310160 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.310124    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a"} err="failed to get container status \"0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a\": rpc error: code = NotFound desc = could not find container \"0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a\": container with ID starting with 0fcd8abab33a3a933b9f5bc3432859754cc6659ca91af601ca3dbc95b0beae0a not found: ID does not exist"
Sep 08 13:45:30.310160 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.310141    2561 scope.go:117] "RemoveContainer" containerID="2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641"
Sep 08 13:45:30.317448 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.317309    2561 scope.go:117] "RemoveContainer" containerID="2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641"
Sep 08 13:45:30.318066 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:45:30.318045    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641\": container with ID starting with 2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641 not found: ID does not exist" containerID="2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641"
Sep 08 13:45:30.318132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.318071    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641"} err="failed to get container status \"2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641\": rpc error: code = NotFound desc = could not find container \"2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641\": container with ID starting with 2bfa67a6237cc845d175395dcecda3937b59b278a120130a1f48dd5cf88a8641 not found: ID does not exist"
Sep 08 13:45:30.319603 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.319586    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"]
Sep 08 13:45:30.323009 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.322989    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fed6eaa3-56ff46c974-jrtwf"]
Sep 08 13:45:30.334876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.334858    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"]
Sep 08 13:45:30.338037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.338015    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fed6eaa3-f568d74b-s2cc6"]
Sep 08 13:45:30.419521 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419493    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"]
Sep 08 13:45:30.419817 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419804    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:30.419866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419819    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:30.419866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419836    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5" containerName="payload-processing"
Sep 08 13:45:30.419866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419842    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5" containerName="payload-processing"
Sep 08 13:45:30.419866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419859    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e801da-a02b-441b-b316-5729c4f4ee16" containerName="revoke-keys"
Sep 08 13:45:30.419866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419865    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e801da-a02b-441b-b316-5729c4f4ee16" containerName="revoke-keys"
Sep 08 13:45:30.420014 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419872    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e1cd6ed-85d0-478b-a0be-8b2ef45464a2" containerName="payload-pre-processing"
Sep 08 13:45:30.420014 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419877    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e1cd6ed-85d0-478b-a0be-8b2ef45464a2" containerName="payload-pre-processing"
Sep 08 13:45:30.420014 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419947    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e1cd6ed-85d0-478b-a0be-8b2ef45464a2" containerName="payload-pre-processing"
Sep 08 13:45:30.420014 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419957    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5" containerName="payload-processing"
Sep 08 13:45:30.420014 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419963    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="250f58a8-dbcb-415b-ad28-bd1c5d653597" containerName="cleanup"
Sep 08 13:45:30.420014 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.419969    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e801da-a02b-441b-b316-5729c4f4ee16" containerName="revoke-keys"
Sep 08 13:45:30.424495 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.424476    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.427704 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.427684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-515aa5b9-openshift-default-dockercfg-klmkw\""
Sep 08 13:45:30.437200 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.437178    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"]
Sep 08 13:45:30.578652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-data\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.578652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7v9\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-kube-api-access-mh7v9\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.578831 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-certs\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.578831 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-envoy\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.578831 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17364275-e171-48b7-8233-9c28c131fb5c-istiod-ca-cert\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.578831 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-istio-token\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.578831 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17364275-e171-48b7-8233-9c28c131fb5c-istio-podinfo\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.578986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-credential-socket\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.578986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.578880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-socket\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7v9\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-kube-api-access-mh7v9\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680305 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-certs\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680305 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-envoy\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680305 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17364275-e171-48b7-8233-9c28c131fb5c-istiod-ca-cert\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680305 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-istio-token\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680569 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17364275-e171-48b7-8233-9c28c131fb5c-istio-podinfo\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680569 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-credential-socket\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680569 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-socket\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680702 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-certs\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680702 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-data\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-credential-socket\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680942 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-socket\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.680942 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-data\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.681042 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.680967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17364275-e171-48b7-8233-9c28c131fb5c-istiod-ca-cert\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.682486 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.682455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-envoy\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.682748 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.682730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17364275-e171-48b7-8233-9c28c131fb5c-istio-podinfo\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.688522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.688498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-istio-token\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.689194 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.689169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7v9\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-kube-api-access-mh7v9\") pod \"e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") " pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:30.733702 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:30.733660    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:31.032966 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:31.032938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"]
Sep 08 13:45:31.034951 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:31.034927    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17364275_e171_48b7_8233_9c28c131fb5c.slice/crio-67ffcc8a604e29e6b734d818a18765f10240c31195e85aef698637cfba250af4 WatchSource:0}: Error finding container 67ffcc8a604e29e6b734d818a18765f10240c31195e85aef698637cfba250af4: Status 404 returned error can't find the container with id 67ffcc8a604e29e6b734d818a18765f10240c31195e85aef698637cfba250af4
Sep 08 13:45:31.037157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:31.037127    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:31.037233 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:31.037191    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:31.037233 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:31.037220    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:31.311647 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:31.311564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" event={"ID":"17364275-e171-48b7-8233-9c28c131fb5c","Type":"ContainerStarted","Data":"b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7"}
Sep 08 13:45:31.311647 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:31.311622    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" event={"ID":"17364275-e171-48b7-8233-9c28c131fb5c","Type":"ContainerStarted","Data":"67ffcc8a604e29e6b734d818a18765f10240c31195e85aef698637cfba250af4"}
Sep 08 13:45:31.333999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:31.333942    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" podStartSLOduration=1.3339180769999999 podStartE2EDuration="1.333918077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:31.331422849 +0000 UTC m=+1039.682408797" watchObservedRunningTime="2026-09-08 13:45:31.333918077 +0000 UTC m=+1039.684904032"
Sep 08 13:45:31.734155 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:31.734126    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:32.245019 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:32.244988    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e1cd6ed-85d0-478b-a0be-8b2ef45464a2" path="/var/lib/kubelet/pods/1e1cd6ed-85d0-478b-a0be-8b2ef45464a2/volumes"
Sep 08 13:45:32.245354 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:32.245342    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5" path="/var/lib/kubelet/pods/ba609fc7-b0b1-46e8-b0c0-0568dc9ddcb5/volumes"
Sep 08 13:45:32.738726 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:32.738696    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:33.124984 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.124914    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"]
Sep 08 13:45:33.129326 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.129309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.132290 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.132268    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b719d2\"/\"kube-root-ca.crt\""
Sep 08 13:45:33.133750 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.132712    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b719d2\"/\"openshift-service-ca.crt\""
Sep 08 13:45:33.133750 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.132770    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b719d2\"/\"autores-model-b719d2-kserve-self-signed-certs\""
Sep 08 13:45:33.138510 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.138480    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b719d2\"/\"default-dockercfg-hd9pv\""
Sep 08 13:45:33.141559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.141408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"]
Sep 08 13:45:33.305347 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.305312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-home\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.305347 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.305347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-dshm\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.305564 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.305400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf45038-58a8-4ffa-9261-e09261fa1ab1-tls-certs\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.305564 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.305443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vjv\" (UniqueName: \"kubernetes.io/projected/baf45038-58a8-4ffa-9261-e09261fa1ab1-kube-api-access-76vjv\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.305564 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.305470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-tmp-dir\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.305564 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.305488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-model-cache\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.320910 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.320881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:33.321815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.321798    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:33.406370 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.406269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-home\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.406370 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.406301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-dshm\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.406370 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.406358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf45038-58a8-4ffa-9261-e09261fa1ab1-tls-certs\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.406691 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.406488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76vjv\" (UniqueName: \"kubernetes.io/projected/baf45038-58a8-4ffa-9261-e09261fa1ab1-kube-api-access-76vjv\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.406691 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.406569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-tmp-dir\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.406691 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.406609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-model-cache\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.407040 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.407014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-tmp-dir\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.407152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.407024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-model-cache\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.407292 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.407271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-home\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.408621 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.408601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-dshm\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.409253 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.409235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf45038-58a8-4ffa-9261-e09261fa1ab1-tls-certs\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.414960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.414941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vjv\" (UniqueName: \"kubernetes.io/projected/baf45038-58a8-4ffa-9261-e09261fa1ab1-kube-api-access-76vjv\") pod \"autores-model-b719d2-kserve-7c94fd57dc-wntqh\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") " pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.444025 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.444001    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:33.725878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:33.725853    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"]
Sep 08 13:45:33.727577 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:33.727550    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf45038_58a8_4ffa_9261_e09261fa1ab1.slice/crio-f31bfe6a849f20865a3a4e463750ce02ee1f93a8b37452620a45ec62d6b788e8 WatchSource:0}: Error finding container f31bfe6a849f20865a3a4e463750ce02ee1f93a8b37452620a45ec62d6b788e8: Status 404 returned error can't find the container with id f31bfe6a849f20865a3a4e463750ce02ee1f93a8b37452620a45ec62d6b788e8
Sep 08 13:45:34.326483 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.326439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh" event={"ID":"baf45038-58a8-4ffa-9261-e09261fa1ab1","Type":"ContainerStarted","Data":"81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484"}
Sep 08 13:45:34.326976 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.326492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh" event={"ID":"baf45038-58a8-4ffa-9261-e09261fa1ab1","Type":"ContainerStarted","Data":"f31bfe6a849f20865a3a4e463750ce02ee1f93a8b37452620a45ec62d6b788e8"}
Sep 08 13:45:34.346926 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.346882    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh" podStartSLOduration=1.3468657450000001 podStartE2EDuration="1.346865745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:34.346165871 +0000 UTC m=+1042.697151818" watchObservedRunningTime="2026-09-08 13:45:34.346865745 +0000 UTC m=+1042.697851693"
Sep 08 13:45:34.531990 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.531953    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"]
Sep 08 13:45:34.535333 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.535311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:34.538188 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.538168    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-515aa5b9\""
Sep 08 13:45:34.546863 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.546836    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"]
Sep 08 13:45:34.574763 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.574735    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"]
Sep 08 13:45:34.577974 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.577930    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:34.580715 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.580697    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-515aa5b9-dockercfg-f8r56\""
Sep 08 13:45:34.590757 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.590733    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"]
Sep 08 13:45:34.618436 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.618403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2k2\" (UniqueName: \"kubernetes.io/projected/2b3a7295-1c21-41f9-8e45-ea4a69663272-kube-api-access-hs2k2\") pod \"payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7\" (UID: \"2b3a7295-1c21-41f9-8e45-ea4a69663272\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:34.618555 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.618494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3a7295-1c21-41f9-8e45-ea4a69663272-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7\" (UID: \"2b3a7295-1c21-41f9-8e45-ea4a69663272\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:34.719613 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.719575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3a7295-1c21-41f9-8e45-ea4a69663272-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7\" (UID: \"2b3a7295-1c21-41f9-8e45-ea4a69663272\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:34.719811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.719625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvnv\" (UniqueName: \"kubernetes.io/projected/14c9aee3-44ec-44be-97fa-c971288d5614-kube-api-access-lcvnv\") pod \"payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv\" (UID: \"14c9aee3-44ec-44be-97fa-c971288d5614\") " pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:34.719811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.719666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c9aee3-44ec-44be-97fa-c971288d5614-plugins-config-volume\") pod \"payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv\" (UID: \"14c9aee3-44ec-44be-97fa-c971288d5614\") " pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:34.719811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.719700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2k2\" (UniqueName: \"kubernetes.io/projected/2b3a7295-1c21-41f9-8e45-ea4a69663272-kube-api-access-hs2k2\") pod \"payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7\" (UID: \"2b3a7295-1c21-41f9-8e45-ea4a69663272\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:34.720212 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.720191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3a7295-1c21-41f9-8e45-ea4a69663272-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7\" (UID: \"2b3a7295-1c21-41f9-8e45-ea4a69663272\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:34.729486 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.729461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2k2\" (UniqueName: \"kubernetes.io/projected/2b3a7295-1c21-41f9-8e45-ea4a69663272-kube-api-access-hs2k2\") pod \"payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7\" (UID: \"2b3a7295-1c21-41f9-8e45-ea4a69663272\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:34.820802 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.820769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvnv\" (UniqueName: \"kubernetes.io/projected/14c9aee3-44ec-44be-97fa-c971288d5614-kube-api-access-lcvnv\") pod \"payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv\" (UID: \"14c9aee3-44ec-44be-97fa-c971288d5614\") " pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:34.820924 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.820818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c9aee3-44ec-44be-97fa-c971288d5614-plugins-config-volume\") pod \"payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv\" (UID: \"14c9aee3-44ec-44be-97fa-c971288d5614\") " pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:34.821567 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.821548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c9aee3-44ec-44be-97fa-c971288d5614-plugins-config-volume\") pod \"payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv\" (UID: \"14c9aee3-44ec-44be-97fa-c971288d5614\") " pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:34.829442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.829400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvnv\" (UniqueName: \"kubernetes.io/projected/14c9aee3-44ec-44be-97fa-c971288d5614-kube-api-access-lcvnv\") pod \"payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv\" (UID: \"14c9aee3-44ec-44be-97fa-c971288d5614\") " pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:34.846355 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.846333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:34.887598 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:34.887562    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:35.021826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.021802    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:45:35.119398 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.119359    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"]
Sep 08 13:45:35.121294 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:35.121272    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3a7295_1c21_41f9_8e45_ea4a69663272.slice/crio-533706c4b08f9c3323f788dd08f18ee8950454b842e3afeb80c225fc97c21a67 WatchSource:0}: Error finding container 533706c4b08f9c3323f788dd08f18ee8950454b842e3afeb80c225fc97c21a67: Status 404 returned error can't find the container with id 533706c4b08f9c3323f788dd08f18ee8950454b842e3afeb80c225fc97c21a67
Sep 08 13:45:35.122746 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.122725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-envoy\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.122853 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.122756    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-istio-token\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.122853 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.122790    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0db05-16c1-42e1-8284-7857a2894753-istiod-ca-cert\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.122853 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.122810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0db05-16c1-42e1-8284-7857a2894753-istio-podinfo\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.122853 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.122825    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-data\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.123177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.122874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-credential-socket\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.123177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.122930    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-certs\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.123177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.122960    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-socket\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.123177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9nw5\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-kube-api-access-g9nw5\") pod \"4ce0db05-16c1-42e1-8284-7857a2894753\" (UID: \"4ce0db05-16c1-42e1-8284-7857a2894753\") "
Sep 08 13:45:35.123362 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:35.123362 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123212    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:35.123362 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123235    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce0db05-16c1-42e1-8284-7857a2894753-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:35.123362 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123313    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce0db05-16c1-42e1-8284-7857a2894753-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.123362 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123331    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.123362 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123343    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.123721 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:35.123721 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.123499    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:35.125105 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.125082    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:35.125183 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.125108    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ce0db05-16c1-42e1-8284-7857a2894753-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:45:35.125250 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.125236    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:35.125414 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.125399    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-kube-api-access-g9nw5" (OuterVolumeSpecName: "kube-api-access-g9nw5") pod "4ce0db05-16c1-42e1-8284-7857a2894753" (UID: "4ce0db05-16c1-42e1-8284-7857a2894753"). InnerVolumeSpecName "kube-api-access-g9nw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:35.167503 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.167481    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"]
Sep 08 13:45:35.168583 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:35.168560    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c9aee3_44ec_44be_97fa_c971288d5614.slice/crio-e1ea9fa2914a1fae26d95ec7beb8e2c26e44844afb8a696a21d4fdf99817aefb WatchSource:0}: Error finding container e1ea9fa2914a1fae26d95ec7beb8e2c26e44844afb8a696a21d4fdf99817aefb: Status 404 returned error can't find the container with id e1ea9fa2914a1fae26d95ec7beb8e2c26e44844afb8a696a21d4fdf99817aefb
Sep 08 13:45:35.224356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.224333    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.224356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.224354    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.224505 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.224363    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce0db05-16c1-42e1-8284-7857a2894753-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.224505 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.224371    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.224505 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.224397    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce0db05-16c1-42e1-8284-7857a2894753-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.224505 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.224405    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9nw5\" (UniqueName: \"kubernetes.io/projected/4ce0db05-16c1-42e1-8284-7857a2894753-kube-api-access-g9nw5\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:45:35.333240 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.333197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7" event={"ID":"2b3a7295-1c21-41f9-8e45-ea4a69663272","Type":"ContainerStarted","Data":"4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d"}
Sep 08 13:45:35.333690 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.333266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7" event={"ID":"2b3a7295-1c21-41f9-8e45-ea4a69663272","Type":"ContainerStarted","Data":"533706c4b08f9c3323f788dd08f18ee8950454b842e3afeb80c225fc97c21a67"}
Sep 08 13:45:35.333690 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.333292    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:35.334840 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.334815    2561 generic.go:356] "Generic (PLEG): container finished" podID="4ce0db05-16c1-42e1-8284-7857a2894753" containerID="c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b" exitCode=0
Sep 08 13:45:35.334979 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.334868    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"
Sep 08 13:45:35.334979 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.334884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" event={"ID":"4ce0db05-16c1-42e1-8284-7857a2894753","Type":"ContainerDied","Data":"c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b"}
Sep 08 13:45:35.334979 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.334932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9" event={"ID":"4ce0db05-16c1-42e1-8284-7857a2894753","Type":"ContainerDied","Data":"0b8519169e44a5437abd342ca13090972add1a29bb4c431b6294cfd98bf9532f"}
Sep 08 13:45:35.334979 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.334956    2561 scope.go:117] "RemoveContainer" containerID="c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b"
Sep 08 13:45:35.337162 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.336655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv" event={"ID":"14c9aee3-44ec-44be-97fa-c971288d5614","Type":"ContainerStarted","Data":"cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25"}
Sep 08 13:45:35.337162 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.336713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv" event={"ID":"14c9aee3-44ec-44be-97fa-c971288d5614","Type":"ContainerStarted","Data":"e1ea9fa2914a1fae26d95ec7beb8e2c26e44844afb8a696a21d4fdf99817aefb"}
Sep 08 13:45:35.337368 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.337222    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:35.348899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.346679    2561 scope.go:117] "RemoveContainer" containerID="a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4"
Sep 08 13:45:35.354948 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.354894    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7" podStartSLOduration=1.354876318 podStartE2EDuration="1.354876318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:35.352996577 +0000 UTC m=+1043.703982520" watchObservedRunningTime="2026-09-08 13:45:35.354876318 +0000 UTC m=+1043.705862267"
Sep 08 13:45:35.358906 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.358887    2561 scope.go:117] "RemoveContainer" containerID="c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b"
Sep 08 13:45:35.359208 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:45:35.359190    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b\": container with ID starting with c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b not found: ID does not exist" containerID="c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b"
Sep 08 13:45:35.359256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.359216    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b"} err="failed to get container status \"c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b\": rpc error: code = NotFound desc = could not find container \"c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b\": container with ID starting with c2689bc675fdb97f4525223e7711b0d7d4ea5f47c8cf85364ecda9d88ef72f2b not found: ID does not exist"
Sep 08 13:45:35.359256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.359233    2561 scope.go:117] "RemoveContainer" containerID="a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4"
Sep 08 13:45:35.359565 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:45:35.359549    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4\": container with ID starting with a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4 not found: ID does not exist" containerID="a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4"
Sep 08 13:45:35.359636 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.359568    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4"} err="failed to get container status \"a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4\": rpc error: code = NotFound desc = could not find container \"a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4\": container with ID starting with a29cf4603eaacb0339caa9287be137e9f7448b0ea58419b0ef4198f7f6308bc4 not found: ID does not exist"
Sep 08 13:45:35.371950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.371813    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv" podStartSLOduration=1.371797919 podStartE2EDuration="1.371797919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:35.371045212 +0000 UTC m=+1043.722031162" watchObservedRunningTime="2026-09-08 13:45:35.371797919 +0000 UTC m=+1043.722783868"
Sep 08 13:45:35.388470 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.388449    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"]
Sep 08 13:45:35.395521 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:35.395499    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fed6eaa3-gw-openshift-default-75d584d64d-hpnw9"]
Sep 08 13:45:36.245566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:36.245530    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" path="/var/lib/kubelet/pods/4ce0db05-16c1-42e1-8284-7857a2894753/volumes"
Sep 08 13:45:43.444171 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:43.444133    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:43.444595 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:43.444190    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:43.456742 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:43.456709    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:44.394002 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:44.393971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:45:46.345749 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:46.345718    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:45:47.351241 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:47.351213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:45:56.449929 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:56.449891    2561 generic.go:356] "Generic (PLEG): container finished" podID="17364275-e171-48b7-8233-9c28c131fb5c" containerID="b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7" exitCode=0
Sep 08 13:45:56.450509 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:56.449972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" event={"ID":"17364275-e171-48b7-8233-9c28c131fb5c","Type":"ContainerDied","Data":"b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7"}
Sep 08 13:45:56.450582 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:56.450540    2561 scope.go:117] "RemoveContainer" containerID="b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7"
Sep 08 13:45:56.451652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:56.451627    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:56.451724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:56.451691    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:56.451761 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:56.451722    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:45:57.461466 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:57.461426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" event={"ID":"17364275-e171-48b7-8233-9c28c131fb5c","Type":"ContainerStarted","Data":"d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88"}
Sep 08 13:45:57.734944 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:57.734788    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:45:57.735134 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:57.735089    2561 patch_prober.go:28] interesting pod/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 08 13:45:57.735202 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:57.735143    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 08 13:45:58.717774 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.717731    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"]
Sep 08 13:45:58.718169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.718050    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy"
Sep 08 13:45:58.718169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.718061    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy"
Sep 08 13:45:58.718169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.718074    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy"
Sep 08 13:45:58.718169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.718079    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy"
Sep 08 13:45:58.718169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.718144    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy"
Sep 08 13:45:58.718169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.718154    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce0db05-16c1-42e1-8284-7857a2894753" containerName="istio-proxy"
Sep 08 13:45:58.721248 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.721226    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"
Sep 08 13:45:58.724318 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.724292    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 13:45:58.724470 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.724293    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 13:45:58.730678 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.730649    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"]
Sep 08 13:45:58.734428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.734365    2561 patch_prober.go:28] interesting pod/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 08 13:45:58.734594 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.734458    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 08 13:45:58.854471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.854417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg\" (UID: \"0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"
Sep 08 13:45:58.955552 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.955499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg\" (UID: \"0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"
Sep 08 13:45:58.956483 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:58.956452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg\" (UID: \"0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"
Sep 08 13:45:59.033344 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:59.033249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"
Sep 08 13:45:59.323544 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:59.323519    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"]
Sep 08 13:45:59.325577 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:45:59.325546    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae2ab76_2cde_4ed2_a1d8_2d922ead6e26.slice/crio-77ef5ef7ccdf5f571cfc62f1d530845d340908996fa96b6d65c6bdeb8dfea0f8 WatchSource:0}: Error finding container 77ef5ef7ccdf5f571cfc62f1d530845d340908996fa96b6d65c6bdeb8dfea0f8: Status 404 returned error can't find the container with id 77ef5ef7ccdf5f571cfc62f1d530845d340908996fa96b6d65c6bdeb8dfea0f8
Sep 08 13:45:59.474371 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:59.474337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg" event={"ID":"0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26","Type":"ContainerStarted","Data":"77ef5ef7ccdf5f571cfc62f1d530845d340908996fa96b6d65c6bdeb8dfea0f8"}
Sep 08 13:45:59.734147 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:59.734110    2561 patch_prober.go:28] interesting pod/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 08 13:45:59.734589 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:45:59.734161    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 08 13:46:00.481566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:00.481535    2561 generic.go:356] "Generic (PLEG): container finished" podID="0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26" containerID="a06735bea3f1c8c992b9fb428a3580be00513681471b16a6b26a15c50e50b848" exitCode=0
Sep 08 13:46:00.481757 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:00.481585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg" event={"ID":"0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26","Type":"ContainerDied","Data":"a06735bea3f1c8c992b9fb428a3580be00513681471b16a6b26a15c50e50b848"}
Sep 08 13:46:00.735325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:00.734872    2561 patch_prober.go:28] interesting pod/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 08 13:46:00.735325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:00.734934    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 08 13:46:01.664990 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:01.664958    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"
Sep 08 13:46:01.735449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:01.734861    2561 patch_prober.go:28] interesting pod/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 08 13:46:01.735449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:01.734932    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 08 13:46:01.784173 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:01.784074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26-maas-api-service-ca\") pod \"0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26\" (UID: \"0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26\") "
Sep 08 13:46:01.784473 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:01.784447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26" (UID: "0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:01.884933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:01.884888    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26-maas-api-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:02.493132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:02.493095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg" event={"ID":"0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26","Type":"ContainerDied","Data":"77ef5ef7ccdf5f571cfc62f1d530845d340908996fa96b6d65c6bdeb8dfea0f8"}
Sep 08 13:46:02.493356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:02.493147    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77ef5ef7ccdf5f571cfc62f1d530845d340908996fa96b6d65c6bdeb8dfea0f8"
Sep 08 13:46:02.493356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:02.493147    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"
Sep 08 13:46:02.738965 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:02.738925    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:46:02.739445 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:02.739313    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:46:02.740093 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:02.740072    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:46:04.519877 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.519843    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"]
Sep 08 13:46:04.520428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.520055    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv" podUID="14c9aee3-44ec-44be-97fa-c971288d5614" containerName="payload-processing" containerID="cri-o://cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25" gracePeriod=30
Sep 08 13:46:04.528135 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.528110    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"]
Sep 08 13:46:04.528369 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.528347    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7" podUID="2b3a7295-1c21-41f9-8e45-ea4a69663272" containerName="payload-pre-processing" containerID="cri-o://4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d" gracePeriod=30
Sep 08 13:46:04.617109 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:04.617067    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3a7295_1c21_41f9_8e45_ea4a69663272.slice/crio-conmon-4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d.scope\": RecentStats: unable to find data in memory cache]"
Sep 08 13:46:04.682629 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.682589    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"]
Sep 08 13:46:04.686016 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.685982    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-515aa5b9-6npfg"]
Sep 08 13:46:04.811943 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.811911    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:46:04.815078 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.815046    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:46:04.911742 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.911686    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c9aee3-44ec-44be-97fa-c971288d5614-plugins-config-volume\") pod \"14c9aee3-44ec-44be-97fa-c971288d5614\" (UID: \"14c9aee3-44ec-44be-97fa-c971288d5614\") "
Sep 08 13:46:04.911958 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.911804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs2k2\" (UniqueName: \"kubernetes.io/projected/2b3a7295-1c21-41f9-8e45-ea4a69663272-kube-api-access-hs2k2\") pod \"2b3a7295-1c21-41f9-8e45-ea4a69663272\" (UID: \"2b3a7295-1c21-41f9-8e45-ea4a69663272\") "
Sep 08 13:46:04.911958 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.911832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcvnv\" (UniqueName: \"kubernetes.io/projected/14c9aee3-44ec-44be-97fa-c971288d5614-kube-api-access-lcvnv\") pod \"14c9aee3-44ec-44be-97fa-c971288d5614\" (UID: \"14c9aee3-44ec-44be-97fa-c971288d5614\") "
Sep 08 13:46:04.911958 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.911870    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3a7295-1c21-41f9-8e45-ea4a69663272-plugins-config-volume\") pod \"2b3a7295-1c21-41f9-8e45-ea4a69663272\" (UID: \"2b3a7295-1c21-41f9-8e45-ea4a69663272\") "
Sep 08 13:46:04.912188 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.912157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c9aee3-44ec-44be-97fa-c971288d5614-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14c9aee3-44ec-44be-97fa-c971288d5614" (UID: "14c9aee3-44ec-44be-97fa-c971288d5614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:04.912301 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.912280    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a7295-1c21-41f9-8e45-ea4a69663272-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b3a7295-1c21-41f9-8e45-ea4a69663272" (UID: "2b3a7295-1c21-41f9-8e45-ea4a69663272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:04.914029 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.914002    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c9aee3-44ec-44be-97fa-c971288d5614-kube-api-access-lcvnv" (OuterVolumeSpecName: "kube-api-access-lcvnv") pod "14c9aee3-44ec-44be-97fa-c971288d5614" (UID: "14c9aee3-44ec-44be-97fa-c971288d5614"). InnerVolumeSpecName "kube-api-access-lcvnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:04.914128 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:04.914033    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3a7295-1c21-41f9-8e45-ea4a69663272-kube-api-access-hs2k2" (OuterVolumeSpecName: "kube-api-access-hs2k2") pod "2b3a7295-1c21-41f9-8e45-ea4a69663272" (UID: "2b3a7295-1c21-41f9-8e45-ea4a69663272"). InnerVolumeSpecName "kube-api-access-hs2k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:05.013329 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.013283    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2k2\" (UniqueName: \"kubernetes.io/projected/2b3a7295-1c21-41f9-8e45-ea4a69663272-kube-api-access-hs2k2\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:05.013329 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.013321    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcvnv\" (UniqueName: \"kubernetes.io/projected/14c9aee3-44ec-44be-97fa-c971288d5614-kube-api-access-lcvnv\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:05.013329 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.013336    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b3a7295-1c21-41f9-8e45-ea4a69663272-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:05.013612 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.013344    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c9aee3-44ec-44be-97fa-c971288d5614-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:05.253000 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.252964    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"]
Sep 08 13:46:05.253287 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253274    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26" containerName="revoke-keys"
Sep 08 13:46:05.253327 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253288    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26" containerName="revoke-keys"
Sep 08 13:46:05.253327 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253307    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a7295-1c21-41f9-8e45-ea4a69663272" containerName="payload-pre-processing"
Sep 08 13:46:05.253327 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253313    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a7295-1c21-41f9-8e45-ea4a69663272" containerName="payload-pre-processing"
Sep 08 13:46:05.253327 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253322    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c9aee3-44ec-44be-97fa-c971288d5614" containerName="payload-processing"
Sep 08 13:46:05.253327 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253327    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c9aee3-44ec-44be-97fa-c971288d5614" containerName="payload-processing"
Sep 08 13:46:05.253520 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253395    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a7295-1c21-41f9-8e45-ea4a69663272" containerName="payload-pre-processing"
Sep 08 13:46:05.253520 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253405    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c9aee3-44ec-44be-97fa-c971288d5614" containerName="payload-processing"
Sep 08 13:46:05.253520 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.253417    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26" containerName="revoke-keys"
Sep 08 13:46:05.256441 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.256419    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"
Sep 08 13:46:05.259538 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.259514    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 13:46:05.259752 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.259737    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 13:46:05.265422 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.265391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"]
Sep 08 13:46:05.315666 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.315619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b719d2-dt8hx\" (UID: \"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"
Sep 08 13:46:05.416872 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.416824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b719d2-dt8hx\" (UID: \"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"
Sep 08 13:46:05.417505 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.417483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b719d2-dt8hx\" (UID: \"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"
Sep 08 13:46:05.511190 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.511100    2561 generic.go:356] "Generic (PLEG): container finished" podID="14c9aee3-44ec-44be-97fa-c971288d5614" containerID="cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25" exitCode=0
Sep 08 13:46:05.511190 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.511145    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"
Sep 08 13:46:05.511479 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.511185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv" event={"ID":"14c9aee3-44ec-44be-97fa-c971288d5614","Type":"ContainerDied","Data":"cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25"}
Sep 08 13:46:05.511479 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.511250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv" event={"ID":"14c9aee3-44ec-44be-97fa-c971288d5614","Type":"ContainerDied","Data":"e1ea9fa2914a1fae26d95ec7beb8e2c26e44844afb8a696a21d4fdf99817aefb"}
Sep 08 13:46:05.511479 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.511279    2561 scope.go:117] "RemoveContainer" containerID="cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25"
Sep 08 13:46:05.512650 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.512623    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b3a7295-1c21-41f9-8e45-ea4a69663272" containerID="4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d" exitCode=0
Sep 08 13:46:05.512788 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.512648    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7" event={"ID":"2b3a7295-1c21-41f9-8e45-ea4a69663272","Type":"ContainerDied","Data":"4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d"}
Sep 08 13:46:05.512788 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.512673    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"
Sep 08 13:46:05.512788 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.512697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7" event={"ID":"2b3a7295-1c21-41f9-8e45-ea4a69663272","Type":"ContainerDied","Data":"533706c4b08f9c3323f788dd08f18ee8950454b842e3afeb80c225fc97c21a67"}
Sep 08 13:46:05.520567 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.520457    2561 scope.go:117] "RemoveContainer" containerID="cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25"
Sep 08 13:46:05.520926 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:05.520838    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25\": container with ID starting with cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25 not found: ID does not exist" containerID="cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25"
Sep 08 13:46:05.520926 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.520880    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25"} err="failed to get container status \"cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25\": rpc error: code = NotFound desc = could not find container \"cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25\": container with ID starting with cd591cfabcd35e3f9351eccdfdd8faaa45bc00020e30623e03bb047799cd9e25 not found: ID does not exist"
Sep 08 13:46:05.520926 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.520897    2561 scope.go:117] "RemoveContainer" containerID="4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d"
Sep 08 13:46:05.529825 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.529801    2561 scope.go:117] "RemoveContainer" containerID="4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d"
Sep 08 13:46:05.530134 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:05.530118    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d\": container with ID starting with 4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d not found: ID does not exist" containerID="4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d"
Sep 08 13:46:05.530188 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.530141    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d"} err="failed to get container status \"4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d\": rpc error: code = NotFound desc = could not find container \"4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d\": container with ID starting with 4a6557b34123758b00e33f1d76904c8f10285d23d5266488c265e599168cf05d not found: ID does not exist"
Sep 08 13:46:05.536541 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.536513    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"]
Sep 08 13:46:05.541028 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.541002    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-515aa5b9-596bd8949c-ks6vv"]
Sep 08 13:46:05.553200 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.553170    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"]
Sep 08 13:46:05.557449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.557423    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-515aa5b9-7f87878d46-9wrd7"]
Sep 08 13:46:05.567711 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.567683    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"
Sep 08 13:46:05.852684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.852661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"]
Sep 08 13:46:05.855318 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:46:05.855282    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f4b4b6_d287_41ae_a8c7_7e574bf29c5d.slice/crio-3ec2cd0ca9b5c56caaa13e50fb3ac75b1d2846372d9aa688647c5e3d9032a9b8 WatchSource:0}: Error finding container 3ec2cd0ca9b5c56caaa13e50fb3ac75b1d2846372d9aa688647c5e3d9032a9b8: Status 404 returned error can't find the container with id 3ec2cd0ca9b5c56caaa13e50fb3ac75b1d2846372d9aa688647c5e3d9032a9b8
Sep 08 13:46:05.908262 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.908227    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"]
Sep 08 13:46:05.908543 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:05.908512    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy" containerID="cri-o://d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88" gracePeriod=30
Sep 08 13:46:06.245831 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.245798    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26" path="/var/lib/kubelet/pods/0ae2ab76-2cde-4ed2-a1d8-2d922ead6e26/volumes"
Sep 08 13:46:06.246173 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.246160    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c9aee3-44ec-44be-97fa-c971288d5614" path="/var/lib/kubelet/pods/14c9aee3-44ec-44be-97fa-c971288d5614/volumes"
Sep 08 13:46:06.246504 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.246490    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b3a7295-1c21-41f9-8e45-ea4a69663272" path="/var/lib/kubelet/pods/2b3a7295-1c21-41f9-8e45-ea4a69663272/volumes"
Sep 08 13:46:06.520566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.520528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx" event={"ID":"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d","Type":"ContainerStarted","Data":"6f3d8f4ea0e26a75b7483b5cf8d6a3e7f361f606ceb4c9f206640a0b3a1ea8aa"}
Sep 08 13:46:06.521007 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.520579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx" event={"ID":"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d","Type":"ContainerStarted","Data":"3ec2cd0ca9b5c56caaa13e50fb3ac75b1d2846372d9aa688647c5e3d9032a9b8"}
Sep 08 13:46:06.539089 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.539030    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx" podStartSLOduration=1.5390148799999999 podStartE2EDuration="1.53901488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:06.537095996 +0000 UTC m=+1074.888081945" watchObservedRunningTime="2026-09-08 13:46:06.53901488 +0000 UTC m=+1074.890000876"
Sep 08 13:46:06.699907 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.699865    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"]
Sep 08 13:46:06.702970 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.702946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.705513 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.705490    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2781f063-openshift-default-dockercfg-7rcq8\""
Sep 08 13:46:06.719650 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.719618    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"]
Sep 08 13:46:06.829228 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-podinfo\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.829228 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829176    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4z9\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-kube-api-access-gg4z9\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.829228 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-socket\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.829558 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2113a-ff32-4238-809b-eb6b4558fdf6-istiod-ca-cert\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.829558 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-data\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.829558 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829407    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-certs\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.829558 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-credential-socket\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.829558 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-envoy\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.829558 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.829535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-token\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.930865 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.930820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-token\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931047 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.930878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-podinfo\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931047 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.930907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4z9\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-kube-api-access-gg4z9\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931047 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.930948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-socket\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931047 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.930978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2113a-ff32-4238-809b-eb6b4558fdf6-istiod-ca-cert\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931047 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-data\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931047 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-certs\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-credential-socket\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-envoy\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931588 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931477    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-socket\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931588 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-credential-socket\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931686 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-certs\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931740 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-data\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.931821 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.931798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2113a-ff32-4238-809b-eb6b4558fdf6-istiod-ca-cert\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.933440 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.933409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-envoy\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.933620 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.933605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-podinfo\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.941126 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.941101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-token\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:06.941277 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:06.941259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4z9\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-kube-api-access-gg4z9\") pod \"e2e-derive-2781f063-openshift-default-6546474756-dqvrf\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") " pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:07.020476 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.020433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:07.298224 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.298198    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"]
Sep 08 13:46:07.300449 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:46:07.300412    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb2113a_ff32_4238_809b_eb6b4558fdf6.slice/crio-12341321593e675644123c4d57318517a6623fc36a32d78576590137d21d841d WatchSource:0}: Error finding container 12341321593e675644123c4d57318517a6623fc36a32d78576590137d21d841d: Status 404 returned error can't find the container with id 12341321593e675644123c4d57318517a6623fc36a32d78576590137d21d841d
Sep 08 13:46:07.302743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.302711    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:46:07.302812 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.302781    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:46:07.302850 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.302810    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:46:07.528202 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.528158    2561 generic.go:356] "Generic (PLEG): container finished" podID="e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d" containerID="6f3d8f4ea0e26a75b7483b5cf8d6a3e7f361f606ceb4c9f206640a0b3a1ea8aa" exitCode=0
Sep 08 13:46:07.528691 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.528250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx" event={"ID":"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d","Type":"ContainerDied","Data":"6f3d8f4ea0e26a75b7483b5cf8d6a3e7f361f606ceb4c9f206640a0b3a1ea8aa"}
Sep 08 13:46:07.530046 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.530021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" event={"ID":"adb2113a-ff32-4238-809b-eb6b4558fdf6","Type":"ContainerStarted","Data":"a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521"}
Sep 08 13:46:07.530212 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.530079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" event={"ID":"adb2113a-ff32-4238-809b-eb6b4558fdf6","Type":"ContainerStarted","Data":"12341321593e675644123c4d57318517a6623fc36a32d78576590137d21d841d"}
Sep 08 13:46:07.571447 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:07.571308    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" podStartSLOduration=1.571285531 podStartE2EDuration="1.571285531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:07.570334935 +0000 UTC m=+1075.921320887" watchObservedRunningTime="2026-09-08 13:46:07.571285531 +0000 UTC m=+1075.922271481"
Sep 08 13:46:08.021275 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:08.021230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:08.026540 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:08.026511    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:08.535451 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:08.535413    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:08.536562 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:08.536537    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:08.677440 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:08.677408    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"
Sep 08 13:46:08.748351 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:08.748306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d-maas-api-service-ca\") pod \"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d\" (UID: \"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d\") "
Sep 08 13:46:08.748741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:08.748715    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d" (UID: "e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:08.849408 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:08.849306    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d-maas-api-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:09.541011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:09.540981    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"
Sep 08 13:46:09.541442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:09.541030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx" event={"ID":"e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d","Type":"ContainerDied","Data":"3ec2cd0ca9b5c56caaa13e50fb3ac75b1d2846372d9aa688647c5e3d9032a9b8"}
Sep 08 13:46:09.541442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:09.541077    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec2cd0ca9b5c56caaa13e50fb3ac75b1d2846372d9aa688647c5e3d9032a9b8"
Sep 08 13:46:10.786098 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.786057    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"]
Sep 08 13:46:10.787790 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.786856    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d" containerName="revoke-keys"
Sep 08 13:46:10.787790 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.786884    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d" containerName="revoke-keys"
Sep 08 13:46:10.787790 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.787120    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d" containerName="revoke-keys"
Sep 08 13:46:10.790815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.790788    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:10.793674 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.793632    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2781f063-dockercfg-scgqp\""
Sep 08 13:46:10.793816 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.793795    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2781f063\""
Sep 08 13:46:10.798011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.797986    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"]
Sep 08 13:46:10.801972 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.801947    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"]
Sep 08 13:46:10.802105 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.802029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:10.818134 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.818107    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"]
Sep 08 13:46:10.868614 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.868573    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wr8g\" (UniqueName: \"kubernetes.io/projected/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-kube-api-access-4wr8g\") pod \"payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh\" (UID: \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:10.868783 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.868708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d95b85-9e58-424d-a2ef-dde1c14c96a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-2781f063-57f967cc97-24cg9\" (UID: \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:10.868783 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.868765    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4mvj\" (UniqueName: \"kubernetes.io/projected/37d95b85-9e58-424d-a2ef-dde1c14c96a1-kube-api-access-q4mvj\") pod \"payload-processing-e2e-derive-2781f063-57f967cc97-24cg9\" (UID: \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:10.868914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.868793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh\" (UID: \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:10.969891 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.969848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4mvj\" (UniqueName: \"kubernetes.io/projected/37d95b85-9e58-424d-a2ef-dde1c14c96a1-kube-api-access-q4mvj\") pod \"payload-processing-e2e-derive-2781f063-57f967cc97-24cg9\" (UID: \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:10.969891 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.969894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh\" (UID: \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:10.970141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.969927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wr8g\" (UniqueName: \"kubernetes.io/projected/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-kube-api-access-4wr8g\") pod \"payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh\" (UID: \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:10.970141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.969982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d95b85-9e58-424d-a2ef-dde1c14c96a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-2781f063-57f967cc97-24cg9\" (UID: \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:10.970759 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.970738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d95b85-9e58-424d-a2ef-dde1c14c96a1-plugins-config-volume\") pod \"payload-processing-e2e-derive-2781f063-57f967cc97-24cg9\" (UID: \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:10.970759 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.970748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh\" (UID: \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:10.978883 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.978858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wr8g\" (UniqueName: \"kubernetes.io/projected/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-kube-api-access-4wr8g\") pod \"payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh\" (UID: \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:10.979016 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:10.978963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4mvj\" (UniqueName: \"kubernetes.io/projected/37d95b85-9e58-424d-a2ef-dde1c14c96a1-kube-api-access-q4mvj\") pod \"payload-processing-e2e-derive-2781f063-57f967cc97-24cg9\" (UID: \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\") " pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:11.103321 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.103290    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:11.111336 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.111305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:11.171151 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.171121    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:46:11.272940 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.272910    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-socket\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.272957    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-envoy\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.272997    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-credential-socket\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273023    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17364275-e171-48b7-8233-9c28c131fb5c-istio-podinfo\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273066    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-data\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273085    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-istio-token\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273115    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17364275-e171-48b7-8233-9c28c131fb5c-istiod-ca-cert\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-certs\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273199    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh7v9\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-kube-api-access-mh7v9\") pod \"17364275-e171-48b7-8233-9c28c131fb5c\" (UID: \"17364275-e171-48b7-8233-9c28c131fb5c\") "
Sep 08 13:46:11.273566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273240    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:11.273566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273420    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:11.273566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:11.273566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273510    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:11.273566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273559    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17364275-e171-48b7-8233-9c28c131fb5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:11.273830 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273581    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.273830 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.273598    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.275340 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.275315    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:11.275472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.275363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17364275-e171-48b7-8233-9c28c131fb5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:46:11.275472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.275368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-kube-api-access-mh7v9" (OuterVolumeSpecName: "kube-api-access-mh7v9") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "kube-api-access-mh7v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:11.275472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.275422    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "17364275-e171-48b7-8233-9c28c131fb5c" (UID: "17364275-e171-48b7-8233-9c28c131fb5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:11.374656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.374559    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17364275-e171-48b7-8233-9c28c131fb5c-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.374656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.374588    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.374656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.374599    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.374656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.374607    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17364275-e171-48b7-8233-9c28c131fb5c-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.374656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.374616    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.374656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.374624    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh7v9\" (UniqueName: \"kubernetes.io/projected/17364275-e171-48b7-8233-9c28c131fb5c-kube-api-access-mh7v9\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.374656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.374633    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17364275-e171-48b7-8233-9c28c131fb5c-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:11.380551 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.380526    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"]
Sep 08 13:46:11.382795 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:46:11.382767    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5d5561_dfb0_43f8_9e96_4b8c98fd6111.slice/crio-22af12331554fe88152215b36fd1b0cccfbc91d958ed0607efc85fed7dab5fdf WatchSource:0}: Error finding container 22af12331554fe88152215b36fd1b0cccfbc91d958ed0607efc85fed7dab5fdf: Status 404 returned error can't find the container with id 22af12331554fe88152215b36fd1b0cccfbc91d958ed0607efc85fed7dab5fdf
Sep 08 13:46:11.406363 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.406333    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"]
Sep 08 13:46:11.416568 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:46:11.416540    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d95b85_9e58_424d_a2ef_dde1c14c96a1.slice/crio-1302d3b54a47d7dea83eb94d9d95bc7f5c7b9140e4762069e93b7fc2574a22c1 WatchSource:0}: Error finding container 1302d3b54a47d7dea83eb94d9d95bc7f5c7b9140e4762069e93b7fc2574a22c1: Status 404 returned error can't find the container with id 1302d3b54a47d7dea83eb94d9d95bc7f5c7b9140e4762069e93b7fc2574a22c1
Sep 08 13:46:11.553841 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.553800    2561 generic.go:356] "Generic (PLEG): container finished" podID="17364275-e171-48b7-8233-9c28c131fb5c" containerID="d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88" exitCode=0
Sep 08 13:46:11.554015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.553847    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"
Sep 08 13:46:11.554015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.553871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" event={"ID":"17364275-e171-48b7-8233-9c28c131fb5c","Type":"ContainerDied","Data":"d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88"}
Sep 08 13:46:11.554015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.553920    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr" event={"ID":"17364275-e171-48b7-8233-9c28c131fb5c","Type":"ContainerDied","Data":"67ffcc8a604e29e6b734d818a18765f10240c31195e85aef698637cfba250af4"}
Sep 08 13:46:11.554015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.553937    2561 scope.go:117] "RemoveContainer" containerID="d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88"
Sep 08 13:46:11.555819 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.555785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh" event={"ID":"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111","Type":"ContainerStarted","Data":"4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e"}
Sep 08 13:46:11.555946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.555834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh" event={"ID":"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111","Type":"ContainerStarted","Data":"22af12331554fe88152215b36fd1b0cccfbc91d958ed0607efc85fed7dab5fdf"}
Sep 08 13:46:11.556021 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.555956    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:11.557605 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.557581    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9" event={"ID":"37d95b85-9e58-424d-a2ef-dde1c14c96a1","Type":"ContainerStarted","Data":"319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4"}
Sep 08 13:46:11.557720 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.557625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9" event={"ID":"37d95b85-9e58-424d-a2ef-dde1c14c96a1","Type":"ContainerStarted","Data":"1302d3b54a47d7dea83eb94d9d95bc7f5c7b9140e4762069e93b7fc2574a22c1"}
Sep 08 13:46:11.557720 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.557701    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:11.564687 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.564671    2561 scope.go:117] "RemoveContainer" containerID="b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7"
Sep 08 13:46:11.573189 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.573172    2561 scope.go:117] "RemoveContainer" containerID="d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88"
Sep 08 13:46:11.573494 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:11.573473    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88\": container with ID starting with d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88 not found: ID does not exist" containerID="d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88"
Sep 08 13:46:11.573543 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.573503    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88"} err="failed to get container status \"d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88\": rpc error: code = NotFound desc = could not find container \"d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88\": container with ID starting with d52ba7e54c809eb442c5c69c61a67bfcba254a17a0db872b3882cbe648ee1a88 not found: ID does not exist"
Sep 08 13:46:11.573543 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.573519    2561 scope.go:117] "RemoveContainer" containerID="b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7"
Sep 08 13:46:11.573807 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:11.573788    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7\": container with ID starting with b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7 not found: ID does not exist" containerID="b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7"
Sep 08 13:46:11.573851 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.573811    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7"} err="failed to get container status \"b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7\": rpc error: code = NotFound desc = could not find container \"b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7\": container with ID starting with b21926b37d8900c27f87eeae901f5dd1c49efa281d3ab87869347e5e6e1a25d7 not found: ID does not exist"
Sep 08 13:46:11.577787 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.577741    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh" podStartSLOduration=1.57772385 podStartE2EDuration="1.57772385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:11.575755281 +0000 UTC m=+1079.926741230" watchObservedRunningTime="2026-09-08 13:46:11.57772385 +0000 UTC m=+1079.928709799"
Sep 08 13:46:11.593253 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.593202    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9" podStartSLOduration=1.593186431 podStartE2EDuration="1.593186431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:11.592089627 +0000 UTC m=+1079.943075576" watchObservedRunningTime="2026-09-08 13:46:11.593186431 +0000 UTC m=+1079.944172378"
Sep 08 13:46:11.605903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.605866    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"]
Sep 08 13:46:11.610282 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:11.610254    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-515aa5b9-openshift-default-6f4cd485fd-f7xwr"]
Sep 08 13:46:12.246866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:12.246826    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17364275-e171-48b7-8233-9c28c131fb5c" path="/var/lib/kubelet/pods/17364275-e171-48b7-8233-9c28c131fb5c/volumes"
Sep 08 13:46:16.002646 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:16.002615    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"]
Sep 08 13:46:16.005929 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:16.005904    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b719d2-29814585-2tvh6"]
Sep 08 13:46:16.221817 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:16.221779    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"]
Sep 08 13:46:16.227033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:16.226999    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b719d2-dt8hx"]
Sep 08 13:46:16.244923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:16.244897    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d" path="/var/lib/kubelet/pods/e2f4b4b6-d287-41ae-a8c7-7e574bf29c5d/volumes"
Sep 08 13:46:16.245179 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:16.245167    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e35380b5-f1e2-4e94-b168-07390ac41a88" path="/var/lib/kubelet/pods/e35380b5-f1e2-4e94-b168-07390ac41a88/volumes"
Sep 08 13:46:21.601195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:21.601156    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"]
Sep 08 13:46:22.565848 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:22.565818    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:22.566051 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:22.565968    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:30.662692 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:30.662657    2561 generic.go:356] "Generic (PLEG): container finished" podID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerID="a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521" exitCode=0
Sep 08 13:46:30.663138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:30.662725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" event={"ID":"adb2113a-ff32-4238-809b-eb6b4558fdf6","Type":"ContainerDied","Data":"a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521"}
Sep 08 13:46:30.663138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:30.663088    2561 scope.go:117] "RemoveContainer" containerID="a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521"
Sep 08 13:46:30.664169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:30.664141    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:46:30.664243 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:30.664229    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:46:30.664286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:30.664266    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:46:31.669214 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:31.669175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" event={"ID":"adb2113a-ff32-4238-809b-eb6b4558fdf6","Type":"ContainerStarted","Data":"8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93"}
Sep 08 13:46:32.021205 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.021104    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:32.021435 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.021361    2561 patch_prober.go:28] interesting pod/e2e-derive-2781f063-openshift-default-6546474756-dqvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 08 13:46:32.021522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.021463    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 08 13:46:32.142440 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.142399    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"]
Sep 08 13:46:32.150695 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.150659    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy"
Sep 08 13:46:32.150935 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.150921    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy"
Sep 08 13:46:32.151073 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.151061    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy"
Sep 08 13:46:32.151196 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.151184    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy"
Sep 08 13:46:32.151425 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.151411    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy"
Sep 08 13:46:32.151534 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.151523    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17364275-e171-48b7-8233-9c28c131fb5c" containerName="istio-proxy"
Sep 08 13:46:32.165622 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.165584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"]
Sep 08 13:46:32.165860 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.165727    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.168534 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.168495    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-c2512ae8-kserve-self-signed-certs\""
Sep 08 13:46:32.269449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.269365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-tmp-dir\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.269643 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.269523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fv98\" (UniqueName: \"kubernetes.io/projected/d4338c7c-29c3-457b-be1d-c73a104a87b7-kube-api-access-6fv98\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.269643 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.269567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4338c7c-29c3-457b-be1d-c73a104a87b7-tls-certs\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.269723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.269648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-model-cache\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.269723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.269682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-dshm\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.269723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.269709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-home\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.370749 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.370634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-tmp-dir\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.370749 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.370722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fv98\" (UniqueName: \"kubernetes.io/projected/d4338c7c-29c3-457b-be1d-c73a104a87b7-kube-api-access-6fv98\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.370749 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.370748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4338c7c-29c3-457b-be1d-c73a104a87b7-tls-certs\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.370997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.370781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-model-cache\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.370997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.370804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-dshm\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.370997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.370826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-home\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.371165 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.371148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-home\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.371348 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.371328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-tmp-dir\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.371551 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.371528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-model-cache\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.373460 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.373436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-dshm\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.373835 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.373819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4338c7c-29c3-457b-be1d-c73a104a87b7-tls-certs\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.380893 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.380868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fv98\" (UniqueName: \"kubernetes.io/projected/d4338c7c-29c3-457b-be1d-c73a104a87b7-kube-api-access-6fv98\") pod \"e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") " pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.475803 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.475762    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:32.749627 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:32.749580    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"]
Sep 08 13:46:33.021312 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:33.021227    2561 patch_prober.go:28] interesting pod/e2e-derive-2781f063-openshift-default-6546474756-dqvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 08 13:46:33.021312 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:33.021287    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 08 13:46:33.680008 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:33.679974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4" event={"ID":"d4338c7c-29c3-457b-be1d-c73a104a87b7","Type":"ContainerStarted","Data":"bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90"}
Sep 08 13:46:33.680008 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:33.680014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4" event={"ID":"d4338c7c-29c3-457b-be1d-c73a104a87b7","Type":"ContainerStarted","Data":"ed662889a1ce4f594f423971b2396a9e42a3a77ba9e7f28ab5a29b419f57ec72"}
Sep 08 13:46:33.697749 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:33.697699    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4" podStartSLOduration=1.697679998 podStartE2EDuration="1.697679998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:33.695960435 +0000 UTC m=+1102.046946396" watchObservedRunningTime="2026-09-08 13:46:33.697679998 +0000 UTC m=+1102.048665946"
Sep 08 13:46:34.021264 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:34.021167    2561 patch_prober.go:28] interesting pod/e2e-derive-2781f063-openshift-default-6546474756-dqvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 08 13:46:34.021264 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:34.021225    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 08 13:46:35.021492 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:35.021455    2561 patch_prober.go:28] interesting pod/e2e-derive-2781f063-openshift-default-6546474756-dqvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 08 13:46:35.021960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:35.021930    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 08 13:46:36.021680 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.021643    2561 patch_prober.go:28] interesting pod/e2e-derive-2781f063-openshift-default-6546474756-dqvrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 08 13:46:36.022077 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.021701    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 08 13:46:36.621826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.621776    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh" podUID="baf45038-58a8-4ffa-9261-e09261fa1ab1" containerName="main" containerID="cri-o://81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484" gracePeriod=45
Sep 08 13:46:36.876777 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.876698    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:46:36.913272 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913222    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-tmp-dir\") pod \"baf45038-58a8-4ffa-9261-e09261fa1ab1\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") "
Sep 08 13:46:36.913272 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913268    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76vjv\" (UniqueName: \"kubernetes.io/projected/baf45038-58a8-4ffa-9261-e09261fa1ab1-kube-api-access-76vjv\") pod \"baf45038-58a8-4ffa-9261-e09261fa1ab1\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") "
Sep 08 13:46:36.913670 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913327    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-dshm\") pod \"baf45038-58a8-4ffa-9261-e09261fa1ab1\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") "
Sep 08 13:46:36.913670 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913365    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-home\") pod \"baf45038-58a8-4ffa-9261-e09261fa1ab1\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") "
Sep 08 13:46:36.913670 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-model-cache\") pod \"baf45038-58a8-4ffa-9261-e09261fa1ab1\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") "
Sep 08 13:46:36.913670 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913487    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf45038-58a8-4ffa-9261-e09261fa1ab1-tls-certs\") pod \"baf45038-58a8-4ffa-9261-e09261fa1ab1\" (UID: \"baf45038-58a8-4ffa-9261-e09261fa1ab1\") "
Sep 08 13:46:36.913670 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913569    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "baf45038-58a8-4ffa-9261-e09261fa1ab1" (UID: "baf45038-58a8-4ffa-9261-e09261fa1ab1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:36.913960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913741    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-home" (OuterVolumeSpecName: "home") pod "baf45038-58a8-4ffa-9261-e09261fa1ab1" (UID: "baf45038-58a8-4ffa-9261-e09261fa1ab1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:36.913960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913846    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:36.913960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913865    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:36.913960 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.913940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-model-cache" (OuterVolumeSpecName: "model-cache") pod "baf45038-58a8-4ffa-9261-e09261fa1ab1" (UID: "baf45038-58a8-4ffa-9261-e09261fa1ab1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:36.916185 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.916151    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf45038-58a8-4ffa-9261-e09261fa1ab1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "baf45038-58a8-4ffa-9261-e09261fa1ab1" (UID: "baf45038-58a8-4ffa-9261-e09261fa1ab1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:46:36.916309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.916183    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-dshm" (OuterVolumeSpecName: "dshm") pod "baf45038-58a8-4ffa-9261-e09261fa1ab1" (UID: "baf45038-58a8-4ffa-9261-e09261fa1ab1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:36.916309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:36.916216    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf45038-58a8-4ffa-9261-e09261fa1ab1-kube-api-access-76vjv" (OuterVolumeSpecName: "kube-api-access-76vjv") pod "baf45038-58a8-4ffa-9261-e09261fa1ab1" (UID: "baf45038-58a8-4ffa-9261-e09261fa1ab1"). InnerVolumeSpecName "kube-api-access-76vjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:37.014988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.014959    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf45038-58a8-4ffa-9261-e09261fa1ab1-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:37.014988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.014986    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76vjv\" (UniqueName: \"kubernetes.io/projected/baf45038-58a8-4ffa-9261-e09261fa1ab1-kube-api-access-76vjv\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:37.014988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.014996    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:37.015214 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.015004    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf45038-58a8-4ffa-9261-e09261fa1ab1-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:37.024723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.024700    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:37.025149 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.024989    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:37.025741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.025721    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:37.542498 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.542465    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"]
Sep 08 13:46:37.542811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.542799    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf45038-58a8-4ffa-9261-e09261fa1ab1" containerName="main"
Sep 08 13:46:37.542859 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.542813    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf45038-58a8-4ffa-9261-e09261fa1ab1" containerName="main"
Sep 08 13:46:37.542894 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.542888    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf45038-58a8-4ffa-9261-e09261fa1ab1" containerName="main"
Sep 08 13:46:37.544769 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.544752    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.547614 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.547596    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2d1742\"/\"kube-root-ca.crt\""
Sep 08 13:46:37.548874 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.548841    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2d1742\"/\"default-dockercfg-6z87x\""
Sep 08 13:46:37.548966 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.548873    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2d1742\"/\"openshift-service-ca.crt\""
Sep 08 13:46:37.548966 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.548861    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2d1742\"/\"test-model-2d1742-kserve-self-signed-certs\""
Sep 08 13:46:37.559442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.559416    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"]
Sep 08 13:46:37.619811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.619774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-dshm\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.619811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.619814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26hk\" (UniqueName: \"kubernetes.io/projected/9865bcc0-80d8-4e67-9467-b77987d8b0fd-kube-api-access-g26hk\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.620024 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.619843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-home\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.620024 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.619869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-model-cache\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.620024 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.619981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tmp-dir\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.620024 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.620003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tls-certs\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.703876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.703468    2561 generic.go:356] "Generic (PLEG): container finished" podID="baf45038-58a8-4ffa-9261-e09261fa1ab1" containerID="81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484" exitCode=0
Sep 08 13:46:37.703876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.703575    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"
Sep 08 13:46:37.703876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.703608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh" event={"ID":"baf45038-58a8-4ffa-9261-e09261fa1ab1","Type":"ContainerDied","Data":"81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484"}
Sep 08 13:46:37.703876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.703640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh" event={"ID":"baf45038-58a8-4ffa-9261-e09261fa1ab1","Type":"ContainerDied","Data":"f31bfe6a849f20865a3a4e463750ce02ee1f93a8b37452620a45ec62d6b788e8"}
Sep 08 13:46:37.703876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.703665    2561 scope.go:117] "RemoveContainer" containerID="81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484"
Sep 08 13:46:37.714045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.714026    2561 scope.go:117] "RemoveContainer" containerID="81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484"
Sep 08 13:46:37.714334 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:37.714316    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484\": container with ID starting with 81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484 not found: ID does not exist" containerID="81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484"
Sep 08 13:46:37.714419 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.714348    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484"} err="failed to get container status \"81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484\": rpc error: code = NotFound desc = could not find container \"81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484\": container with ID starting with 81a495433cc60f7311757e208573be6b8a1f6f82befe9c53a9c309915185f484 not found: ID does not exist"
Sep 08 13:46:37.720885 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.720864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tmp-dir\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.720989 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.720916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tls-certs\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.721060 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.720995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-dshm\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.721060 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.721020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26hk\" (UniqueName: \"kubernetes.io/projected/9865bcc0-80d8-4e67-9467-b77987d8b0fd-kube-api-access-g26hk\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.721164 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.721061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-home\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.721164 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.721117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-model-cache\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.721275 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.721227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tmp-dir\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.721446 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.721422    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-model-cache\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.721554 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.721466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-home\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.723200 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.723178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-dshm\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.723394 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.723366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tls-certs\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.728336 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.728312    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"]
Sep 08 13:46:37.731359 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.731337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26hk\" (UniqueName: \"kubernetes.io/projected/9865bcc0-80d8-4e67-9467-b77987d8b0fd-kube-api-access-g26hk\") pod \"test-model-2d1742-kserve-86995fdf57-8z8xn\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") " pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:37.732797 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.732776    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b719d2/autores-model-b719d2-kserve-7c94fd57dc-wntqh"]
Sep 08 13:46:37.854659 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:37.854569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:38.125041 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:38.124828    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"]
Sep 08 13:46:38.127820 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:46:38.127793    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9865bcc0_80d8_4e67_9467_b77987d8b0fd.slice/crio-939959d157a879ccc71a29458373c8ee734bae26c5ed35928fa09bdc957ce236 WatchSource:0}: Error finding container 939959d157a879ccc71a29458373c8ee734bae26c5ed35928fa09bdc957ce236: Status 404 returned error can't find the container with id 939959d157a879ccc71a29458373c8ee734bae26c5ed35928fa09bdc957ce236
Sep 08 13:46:38.245986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:38.245941    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf45038-58a8-4ffa-9261-e09261fa1ab1" path="/var/lib/kubelet/pods/baf45038-58a8-4ffa-9261-e09261fa1ab1/volumes"
Sep 08 13:46:38.710500 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:38.710461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn" event={"ID":"9865bcc0-80d8-4e67-9467-b77987d8b0fd","Type":"ContainerStarted","Data":"ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb"}
Sep 08 13:46:38.710703 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:38.710509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn" event={"ID":"9865bcc0-80d8-4e67-9467-b77987d8b0fd","Type":"ContainerStarted","Data":"939959d157a879ccc71a29458373c8ee734bae26c5ed35928fa09bdc957ce236"}
Sep 08 13:46:38.737417 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:38.737341    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn" podStartSLOduration=1.737319633 podStartE2EDuration="1.737319633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:38.735658168 +0000 UTC m=+1107.086644137" watchObservedRunningTime="2026-09-08 13:46:38.737319633 +0000 UTC m=+1107.088305581"
Sep 08 13:46:40.779710 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:40.779674    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"]
Sep 08 13:46:40.780202 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:40.779900    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9" podUID="37d95b85-9e58-424d-a2ef-dde1c14c96a1" containerName="payload-processing" containerID="cri-o://319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4" gracePeriod=30
Sep 08 13:46:40.785842 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:40.785804    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"]
Sep 08 13:46:40.787037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:40.786983    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh" podUID="9b5d5561-dfb0-43f8-9e96-4b8c98fd6111" containerName="payload-pre-processing" containerID="cri-o://4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e" gracePeriod=30
Sep 08 13:46:41.051427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.051399    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:41.054284 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.054264    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:41.127809 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.127773    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"]
Sep 08 13:46:41.128001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.127985    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy" containerID="cri-o://8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93" gracePeriod=30
Sep 08 13:46:41.148703 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.148675    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wr8g\" (UniqueName: \"kubernetes.io/projected/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-kube-api-access-4wr8g\") pod \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\" (UID: \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\") "
Sep 08 13:46:41.148855 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.148710    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-plugins-config-volume\") pod \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\" (UID: \"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111\") "
Sep 08 13:46:41.148855 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.148747    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4mvj\" (UniqueName: \"kubernetes.io/projected/37d95b85-9e58-424d-a2ef-dde1c14c96a1-kube-api-access-q4mvj\") pod \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\" (UID: \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\") "
Sep 08 13:46:41.148855 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.148776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d95b85-9e58-424d-a2ef-dde1c14c96a1-plugins-config-volume\") pod \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\" (UID: \"37d95b85-9e58-424d-a2ef-dde1c14c96a1\") "
Sep 08 13:46:41.149195 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.149164    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b5d5561-dfb0-43f8-9e96-4b8c98fd6111" (UID: "9b5d5561-dfb0-43f8-9e96-4b8c98fd6111"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:41.149862 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.149233    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d95b85-9e58-424d-a2ef-dde1c14c96a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37d95b85-9e58-424d-a2ef-dde1c14c96a1" (UID: "37d95b85-9e58-424d-a2ef-dde1c14c96a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:41.151130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.151103    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d95b85-9e58-424d-a2ef-dde1c14c96a1-kube-api-access-q4mvj" (OuterVolumeSpecName: "kube-api-access-q4mvj") pod "37d95b85-9e58-424d-a2ef-dde1c14c96a1" (UID: "37d95b85-9e58-424d-a2ef-dde1c14c96a1"). InnerVolumeSpecName "kube-api-access-q4mvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:41.151225 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.151185    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-kube-api-access-4wr8g" (OuterVolumeSpecName: "kube-api-access-4wr8g") pod "9b5d5561-dfb0-43f8-9e96-4b8c98fd6111" (UID: "9b5d5561-dfb0-43f8-9e96-4b8c98fd6111"). InnerVolumeSpecName "kube-api-access-4wr8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:41.250546 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.250507    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wr8g\" (UniqueName: \"kubernetes.io/projected/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-kube-api-access-4wr8g\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:41.250546 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.250540    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:41.250546 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.250552    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4mvj\" (UniqueName: \"kubernetes.io/projected/37d95b85-9e58-424d-a2ef-dde1c14c96a1-kube-api-access-q4mvj\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:41.250836 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.250564    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d95b85-9e58-424d-a2ef-dde1c14c96a1-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:41.737623 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.737586    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b5d5561-dfb0-43f8-9e96-4b8c98fd6111" containerID="4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e" exitCode=0
Sep 08 13:46:41.737800 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.737637    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"
Sep 08 13:46:41.737800 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.737671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh" event={"ID":"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111","Type":"ContainerDied","Data":"4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e"}
Sep 08 13:46:41.737800 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.737729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh" event={"ID":"9b5d5561-dfb0-43f8-9e96-4b8c98fd6111","Type":"ContainerDied","Data":"22af12331554fe88152215b36fd1b0cccfbc91d958ed0607efc85fed7dab5fdf"}
Sep 08 13:46:41.737800 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.737761    2561 scope.go:117] "RemoveContainer" containerID="4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e"
Sep 08 13:46:41.739035 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.739009    2561 generic.go:356] "Generic (PLEG): container finished" podID="37d95b85-9e58-424d-a2ef-dde1c14c96a1" containerID="319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4" exitCode=0
Sep 08 13:46:41.739138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.739038    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9" event={"ID":"37d95b85-9e58-424d-a2ef-dde1c14c96a1","Type":"ContainerDied","Data":"319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4"}
Sep 08 13:46:41.739138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.739059    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"
Sep 08 13:46:41.739138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.739080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9" event={"ID":"37d95b85-9e58-424d-a2ef-dde1c14c96a1","Type":"ContainerDied","Data":"1302d3b54a47d7dea83eb94d9d95bc7f5c7b9140e4762069e93b7fc2574a22c1"}
Sep 08 13:46:41.748682 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.748660    2561 scope.go:117] "RemoveContainer" containerID="4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e"
Sep 08 13:46:41.749081 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:41.749056    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e\": container with ID starting with 4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e not found: ID does not exist" containerID="4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e"
Sep 08 13:46:41.749167 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.749092    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e"} err="failed to get container status \"4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e\": rpc error: code = NotFound desc = could not find container \"4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e\": container with ID starting with 4abbe4e62b4e032f3c406e7e4004f98967a087210e00a98285c8c16886f9934e not found: ID does not exist"
Sep 08 13:46:41.749167 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.749113    2561 scope.go:117] "RemoveContainer" containerID="319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4"
Sep 08 13:46:41.757664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.757640    2561 scope.go:117] "RemoveContainer" containerID="319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4"
Sep 08 13:46:41.757994 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:41.757976    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4\": container with ID starting with 319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4 not found: ID does not exist" containerID="319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4"
Sep 08 13:46:41.758036 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.758002    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4"} err="failed to get container status \"319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4\": rpc error: code = NotFound desc = could not find container \"319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4\": container with ID starting with 319853e98bdcddd2546987a274dc864a891ace8748e6e28f8fa1595454b206a4 not found: ID does not exist"
Sep 08 13:46:41.767676 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.767619    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"]
Sep 08 13:46:41.771369 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.771313    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2781f063-d7c6b76c8-dmkbh"]
Sep 08 13:46:41.783277 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.783243    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"]
Sep 08 13:46:41.788201 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:41.788162    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2781f063-57f967cc97-24cg9"]
Sep 08 13:46:42.245397 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:42.245360    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d95b85-9e58-424d-a2ef-dde1c14c96a1" path="/var/lib/kubelet/pods/37d95b85-9e58-424d-a2ef-dde1c14c96a1/volumes"
Sep 08 13:46:42.245751 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:42.245738    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b5d5561-dfb0-43f8-9e96-4b8c98fd6111" path="/var/lib/kubelet/pods/9b5d5561-dfb0-43f8-9e96-4b8c98fd6111/volumes"
Sep 08 13:46:42.476176 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:42.476123    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:42.476460 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:42.476206    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:42.489004 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:42.488975    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:42.757324 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:42.757295    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:46:44.520068 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:44.520031    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"]
Sep 08 13:46:46.375885 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.375855    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:46.397158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-token\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397132    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg4z9\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-kube-api-access-gg4z9\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397153    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-certs\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397177    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-podinfo\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-socket\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-envoy\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397301    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-credential-socket\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2113a-ff32-4238-809b-eb6b4558fdf6-istiod-ca-cert\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-data\") pod \"adb2113a-ff32-4238-809b-eb6b4558fdf6\" (UID: \"adb2113a-ff32-4238-809b-eb6b4558fdf6\") "
Sep 08 13:46:46.397773 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:46.397773 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397702    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.397773 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397721    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-data" (OuterVolumeSpecName: "istio-data") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:46.398040 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.397967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb2113a-ff32-4238-809b-eb6b4558fdf6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:46:46.398301 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.398261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:46.398586 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.398555    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:46.399587 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.399563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-token" (OuterVolumeSpecName: "istio-token") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:46.400085 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.400060    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:46.400163 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.400123    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:46:46.400297 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.400272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-kube-api-access-gg4z9" (OuterVolumeSpecName: "kube-api-access-gg4z9") pod "adb2113a-ff32-4238-809b-eb6b4558fdf6" (UID: "adb2113a-ff32-4238-809b-eb6b4558fdf6"). InnerVolumeSpecName "kube-api-access-gg4z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:46.498797 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.498762    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.498797 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.498792    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.498797 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.498801    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4z9\" (UniqueName: \"kubernetes.io/projected/adb2113a-ff32-4238-809b-eb6b4558fdf6-kube-api-access-gg4z9\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.499032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.498810    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.499032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.498819    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.499032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.498827    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.499032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.498836    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb2113a-ff32-4238-809b-eb6b4558fdf6-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.499032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.498844    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb2113a-ff32-4238-809b-eb6b4558fdf6-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:46:46.770946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.770901    2561 generic.go:356] "Generic (PLEG): container finished" podID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerID="8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93" exitCode=0
Sep 08 13:46:46.771128 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.770950    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"
Sep 08 13:46:46.771128 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.770993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" event={"ID":"adb2113a-ff32-4238-809b-eb6b4558fdf6","Type":"ContainerDied","Data":"8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93"}
Sep 08 13:46:46.771128 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.771057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf" event={"ID":"adb2113a-ff32-4238-809b-eb6b4558fdf6","Type":"ContainerDied","Data":"12341321593e675644123c4d57318517a6623fc36a32d78576590137d21d841d"}
Sep 08 13:46:46.771128 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.771092    2561 scope.go:117] "RemoveContainer" containerID="8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93"
Sep 08 13:46:46.781323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.781301    2561 scope.go:117] "RemoveContainer" containerID="a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521"
Sep 08 13:46:46.789708 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.789690    2561 scope.go:117] "RemoveContainer" containerID="8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93"
Sep 08 13:46:46.790002 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:46.789980    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93\": container with ID starting with 8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93 not found: ID does not exist" containerID="8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93"
Sep 08 13:46:46.790044 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.790013    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93"} err="failed to get container status \"8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93\": rpc error: code = NotFound desc = could not find container \"8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93\": container with ID starting with 8ffa3c76cf00369c5f858677bacfe6a4e5b3113b9859cc8073ae028151d39f93 not found: ID does not exist"
Sep 08 13:46:46.790044 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.790030    2561 scope.go:117] "RemoveContainer" containerID="a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521"
Sep 08 13:46:46.790303 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:46:46.790286    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521\": container with ID starting with a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521 not found: ID does not exist" containerID="a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521"
Sep 08 13:46:46.790344 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.790309    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521"} err="failed to get container status \"a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521\": rpc error: code = NotFound desc = could not find container \"a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521\": container with ID starting with a71cbeed79cac8761ead8d17cbe2acd0fb9fbbbe90024e641fd6638a4a065521 not found: ID does not exist"
Sep 08 13:46:46.797998 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.797960    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"]
Sep 08 13:46:46.805660 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:46.805629    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2781f063-openshift-default-6546474756-dqvrf"]
Sep 08 13:46:47.855163 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:47.855109    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:47.855163 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:47.855172    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:47.868074 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:47.868047    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:48.244491 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:48.244449    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" path="/var/lib/kubelet/pods/adb2113a-ff32-4238-809b-eb6b4558fdf6/volumes"
Sep 08 13:46:48.794905 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:48.794875    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:46:55.732954 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:55.732913    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"]
Sep 08 13:46:59.015713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.015678    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"]
Sep 08 13:46:59.016240 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.015903    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy" containerID="cri-o://67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3" gracePeriod=30
Sep 08 13:46:59.775966 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.775896    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4" podUID="d4338c7c-29c3-457b-be1d-c73a104a87b7" containerName="main" containerID="cri-o://bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90" gracePeriod=45
Sep 08 13:46:59.834811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.834779    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"]
Sep 08 13:46:59.835268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835247    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b5d5561-dfb0-43f8-9e96-4b8c98fd6111" containerName="payload-pre-processing"
Sep 08 13:46:59.835320 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835272    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b5d5561-dfb0-43f8-9e96-4b8c98fd6111" containerName="payload-pre-processing"
Sep 08 13:46:59.835320 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835298    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy"
Sep 08 13:46:59.835320 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835307    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy"
Sep 08 13:46:59.835440 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835321    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy"
Sep 08 13:46:59.835440 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835330    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy"
Sep 08 13:46:59.835440 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835352    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d95b85-9e58-424d-a2ef-dde1c14c96a1" containerName="payload-processing"
Sep 08 13:46:59.835440 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835360    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d95b85-9e58-424d-a2ef-dde1c14c96a1" containerName="payload-processing"
Sep 08 13:46:59.835557 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835490    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d95b85-9e58-424d-a2ef-dde1c14c96a1" containerName="payload-processing"
Sep 08 13:46:59.835557 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835508    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b5d5561-dfb0-43f8-9e96-4b8c98fd6111" containerName="payload-pre-processing"
Sep 08 13:46:59.835557 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835523    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy"
Sep 08 13:46:59.835557 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.835540    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb2113a-ff32-4238-809b-eb6b4558fdf6" containerName="istio-proxy"
Sep 08 13:46:59.838771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.838751    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.841878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.841859    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-d60b5a-openshift-default-dockercfg-g8kvt\""
Sep 08 13:46:59.849985 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.849949    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"]
Sep 08 13:46:59.909139 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-certs\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.909139 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-credential-socket\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.909372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbw6w\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-kube-api-access-lbw6w\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.909372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-envoy\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.909372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-data\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.909527 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-socket\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.909527 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909429    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90523b1d-fc1f-4795-aacc-5335435256cf-istio-podinfo\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.909527 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-istio-token\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:46:59.909527 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:46:59.909509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90523b1d-fc1f-4795-aacc-5335435256cf-istiod-ca-cert\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-socket\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010651 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90523b1d-fc1f-4795-aacc-5335435256cf-istio-podinfo\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010651 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010498    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-istio-token\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010651 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90523b1d-fc1f-4795-aacc-5335435256cf-istiod-ca-cert\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-certs\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-credential-socket\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbw6w\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-kube-api-access-lbw6w\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-envoy\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-socket\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.010930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.010914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-data\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.011261 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.011140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-credential-socket\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.011261 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.011179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-certs\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.011261 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.011218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-data\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.011752 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.011729    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90523b1d-fc1f-4795-aacc-5335435256cf-istiod-ca-cert\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.013392 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.013353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-envoy\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.013567 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.013549    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90523b1d-fc1f-4795-aacc-5335435256cf-istio-podinfo\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.019786 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.019758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-istio-token\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.020109 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.019809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbw6w\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-kube-api-access-lbw6w\") pod \"e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") " pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.040891 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.040804    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:47:00.111278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111238    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4338c7c-29c3-457b-be1d-c73a104a87b7-tls-certs\") pod \"d4338c7c-29c3-457b-be1d-c73a104a87b7\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") "
Sep 08 13:47:00.111547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-tmp-dir\") pod \"d4338c7c-29c3-457b-be1d-c73a104a87b7\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") "
Sep 08 13:47:00.111547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111323    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-home\") pod \"d4338c7c-29c3-457b-be1d-c73a104a87b7\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") "
Sep 08 13:47:00.111547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111340    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-dshm\") pod \"d4338c7c-29c3-457b-be1d-c73a104a87b7\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") "
Sep 08 13:47:00.111547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fv98\" (UniqueName: \"kubernetes.io/projected/d4338c7c-29c3-457b-be1d-c73a104a87b7-kube-api-access-6fv98\") pod \"d4338c7c-29c3-457b-be1d-c73a104a87b7\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") "
Sep 08 13:47:00.111547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111411    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-model-cache\") pod \"d4338c7c-29c3-457b-be1d-c73a104a87b7\" (UID: \"d4338c7c-29c3-457b-be1d-c73a104a87b7\") "
Sep 08 13:47:00.111826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111643    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4338c7c-29c3-457b-be1d-c73a104a87b7" (UID: "d4338c7c-29c3-457b-be1d-c73a104a87b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:00.111826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111655    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-home" (OuterVolumeSpecName: "home") pod "d4338c7c-29c3-457b-be1d-c73a104a87b7" (UID: "d4338c7c-29c3-457b-be1d-c73a104a87b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:00.111826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.111787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4338c7c-29c3-457b-be1d-c73a104a87b7" (UID: "d4338c7c-29c3-457b-be1d-c73a104a87b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:00.113592 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.113561    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-dshm" (OuterVolumeSpecName: "dshm") pod "d4338c7c-29c3-457b-be1d-c73a104a87b7" (UID: "d4338c7c-29c3-457b-be1d-c73a104a87b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:00.113697 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.113643    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4338c7c-29c3-457b-be1d-c73a104a87b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4338c7c-29c3-457b-be1d-c73a104a87b7" (UID: "d4338c7c-29c3-457b-be1d-c73a104a87b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:00.113697 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.113690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4338c7c-29c3-457b-be1d-c73a104a87b7-kube-api-access-6fv98" (OuterVolumeSpecName: "kube-api-access-6fv98") pod "d4338c7c-29c3-457b-be1d-c73a104a87b7" (UID: "d4338c7c-29c3-457b-be1d-c73a104a87b7"). InnerVolumeSpecName "kube-api-access-6fv98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:00.165980 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.165923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:00.212704 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.212670    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4338c7c-29c3-457b-be1d-c73a104a87b7-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:00.212704 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.212702    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:00.212704 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.212712    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:00.213124 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.212719    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:00.213124 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.212729    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fv98\" (UniqueName: \"kubernetes.io/projected/d4338c7c-29c3-457b-be1d-c73a104a87b7-kube-api-access-6fv98\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:00.213124 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.212741    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4338c7c-29c3-457b-be1d-c73a104a87b7-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:00.441997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.441971    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"]
Sep 08 13:47:00.444277 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:47:00.444246    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90523b1d_fc1f_4795_aacc_5335435256cf.slice/crio-6a7e1ccacd56e3877f5334d94b7f5ee9153341cf9d54811ffc4f24ede3fc6cac WatchSource:0}: Error finding container 6a7e1ccacd56e3877f5334d94b7f5ee9153341cf9d54811ffc4f24ede3fc6cac: Status 404 returned error can't find the container with id 6a7e1ccacd56e3877f5334d94b7f5ee9153341cf9d54811ffc4f24ede3fc6cac
Sep 08 13:47:00.446352 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.446324    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:47:00.446428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.446407    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:47:00.446468 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.446437    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:47:00.846777 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.846687    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4338c7c-29c3-457b-be1d-c73a104a87b7" containerID="bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90" exitCode=0
Sep 08 13:47:00.846777 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.846741    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"
Sep 08 13:47:00.846777 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.846760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4" event={"ID":"d4338c7c-29c3-457b-be1d-c73a104a87b7","Type":"ContainerDied","Data":"bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90"}
Sep 08 13:47:00.847070 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.846805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4" event={"ID":"d4338c7c-29c3-457b-be1d-c73a104a87b7","Type":"ContainerDied","Data":"ed662889a1ce4f594f423971b2396a9e42a3a77ba9e7f28ab5a29b419f57ec72"}
Sep 08 13:47:00.847070 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.846827    2561 scope.go:117] "RemoveContainer" containerID="bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90"
Sep 08 13:47:00.848721 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.848585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" event={"ID":"90523b1d-fc1f-4795-aacc-5335435256cf","Type":"ContainerStarted","Data":"e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1"}
Sep 08 13:47:00.848721 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.848639    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" event={"ID":"90523b1d-fc1f-4795-aacc-5335435256cf","Type":"ContainerStarted","Data":"6a7e1ccacd56e3877f5334d94b7f5ee9153341cf9d54811ffc4f24ede3fc6cac"}
Sep 08 13:47:00.855782 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.855762    2561 scope.go:117] "RemoveContainer" containerID="bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90"
Sep 08 13:47:00.856109 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:47:00.856082    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90\": container with ID starting with bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90 not found: ID does not exist" containerID="bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90"
Sep 08 13:47:00.856158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.856119    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90"} err="failed to get container status \"bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90\": rpc error: code = NotFound desc = could not find container \"bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90\": container with ID starting with bf83e7220b48a40c8eb27cbb57ee14159e00de16ca49c550f79c09657d3c2d90 not found: ID does not exist"
Sep 08 13:47:00.865633 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.865588    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"]
Sep 08 13:47:00.870156 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.870126    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-c2512ae8-kserve-664d9b68db-msnp4"]
Sep 08 13:47:00.889890 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:00.889836    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" podStartSLOduration=1.8898183290000001 podStartE2EDuration="1.889818329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:00.888358641 +0000 UTC m=+1129.239344591" watchObservedRunningTime="2026-09-08 13:47:00.889818329 +0000 UTC m=+1129.240804276"
Sep 08 13:47:01.166502 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:01.166412    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:01.171569 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:01.171538    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:01.854841 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:01.854805    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:01.855814 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:01.855791    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:02.245490 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:02.245457    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4338c7c-29c3-457b-be1d-c73a104a87b7" path="/var/lib/kubelet/pods/d4338c7c-29c3-457b-be1d-c73a104a87b7/volumes"
Sep 08 13:47:04.271546 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.271516    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:47:04.352998 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.352902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-certs\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.352998 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.352958    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-credential-socket\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.353237 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-token\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.353237 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353038    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-envoy\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.353237 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-data\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.353237 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrlhl\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-kube-api-access-hrlhl\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.353237 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353171    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-podinfo\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.353547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:04.353547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353319    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-socket\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.353547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353325    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:04.353547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353399    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26bcd53-9ee2-4082-809b-b28852380d9a-istiod-ca-cert\") pod \"c26bcd53-9ee2-4082-809b-b28852380d9a\" (UID: \"c26bcd53-9ee2-4082-809b-b28852380d9a\") "
Sep 08 13:47:04.353772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353569    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:04.353772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:04.353772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26bcd53-9ee2-4082-809b-b28852380d9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:04.353930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353798    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.353930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353817    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.353930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353831    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.353930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.353846    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.355486 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.355455    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:04.355593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.355525    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:04.355593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.355541    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:47:04.355593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.355568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-kube-api-access-hrlhl" (OuterVolumeSpecName: "kube-api-access-hrlhl") pod "c26bcd53-9ee2-4082-809b-b28852380d9a" (UID: "c26bcd53-9ee2-4082-809b-b28852380d9a"). InnerVolumeSpecName "kube-api-access-hrlhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:04.454981 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.454936    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26bcd53-9ee2-4082-809b-b28852380d9a-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.454981 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.454973    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.454981 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.454986    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.454981 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.454993    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrlhl\" (UniqueName: \"kubernetes.io/projected/c26bcd53-9ee2-4082-809b-b28852380d9a-kube-api-access-hrlhl\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.455266 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.455002    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26bcd53-9ee2-4082-809b-b28852380d9a-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:04.700792 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.700757    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"]
Sep 08 13:47:04.701086 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701073    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy"
Sep 08 13:47:04.701133 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701088    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy"
Sep 08 13:47:04.701133 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701101    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy"
Sep 08 13:47:04.701133 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701107    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy"
Sep 08 13:47:04.701133 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701115    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4338c7c-29c3-457b-be1d-c73a104a87b7" containerName="main"
Sep 08 13:47:04.701133 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701121    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4338c7c-29c3-457b-be1d-c73a104a87b7" containerName="main"
Sep 08 13:47:04.701288 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701172    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4338c7c-29c3-457b-be1d-c73a104a87b7" containerName="main"
Sep 08 13:47:04.701288 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701178    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy"
Sep 08 13:47:04.701288 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.701186    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerName="istio-proxy"
Sep 08 13:47:04.704309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.704292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:04.707137 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.707107    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-d60b5a\""
Sep 08 13:47:04.708421 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.708402    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-d60b5a-dockercfg-2s9qh\""
Sep 08 13:47:04.720615 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.720584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"]
Sep 08 13:47:04.757111 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.757076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ed378e-38c7-4f27-b79a-117601145a68-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm\" (UID: \"59ed378e-38c7-4f27-b79a-117601145a68\") " pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:04.757282 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.757130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/59ed378e-38c7-4f27-b79a-117601145a68-kube-api-access-bwgkm\") pod \"payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm\" (UID: \"59ed378e-38c7-4f27-b79a-117601145a68\") " pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:04.757633 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.757612    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"]
Sep 08 13:47:04.760668 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.760651    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:04.771021 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.770994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"]
Sep 08 13:47:04.858300 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.858262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ed378e-38c7-4f27-b79a-117601145a68-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm\" (UID: \"59ed378e-38c7-4f27-b79a-117601145a68\") " pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:04.858522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.858309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97csm\" (UniqueName: \"kubernetes.io/projected/ecaedac2-3f93-49fa-bba4-d5a674894361-kube-api-access-97csm\") pod \"payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6\" (UID: \"ecaedac2-3f93-49fa-bba4-d5a674894361\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:04.858522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.858339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaedac2-3f93-49fa-bba4-d5a674894361-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6\" (UID: \"ecaedac2-3f93-49fa-bba4-d5a674894361\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:04.858522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.858424    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/59ed378e-38c7-4f27-b79a-117601145a68-kube-api-access-bwgkm\") pod \"payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm\" (UID: \"59ed378e-38c7-4f27-b79a-117601145a68\") " pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:04.859018 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.858996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ed378e-38c7-4f27-b79a-117601145a68-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm\" (UID: \"59ed378e-38c7-4f27-b79a-117601145a68\") " pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:04.866873 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.866847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/59ed378e-38c7-4f27-b79a-117601145a68-kube-api-access-bwgkm\") pod \"payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm\" (UID: \"59ed378e-38c7-4f27-b79a-117601145a68\") " pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:04.870446 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.870416    2561 generic.go:356] "Generic (PLEG): container finished" podID="c26bcd53-9ee2-4082-809b-b28852380d9a" containerID="67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3" exitCode=0
Sep 08 13:47:04.870594 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.870462    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"
Sep 08 13:47:04.870594 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.870486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" event={"ID":"c26bcd53-9ee2-4082-809b-b28852380d9a","Type":"ContainerDied","Data":"67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3"}
Sep 08 13:47:04.870594 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.870544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg" event={"ID":"c26bcd53-9ee2-4082-809b-b28852380d9a","Type":"ContainerDied","Data":"77cc13750885408fa2c164c515ee1d3ce1b0c6f496c13ac0272599ea1f027749"}
Sep 08 13:47:04.870594 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.870573    2561 scope.go:117] "RemoveContainer" containerID="67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3"
Sep 08 13:47:04.880192 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.880164    2561 scope.go:117] "RemoveContainer" containerID="da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7"
Sep 08 13:47:04.889022 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.889002    2561 scope.go:117] "RemoveContainer" containerID="67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3"
Sep 08 13:47:04.889355 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:47:04.889335    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3\": container with ID starting with 67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3 not found: ID does not exist" containerID="67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3"
Sep 08 13:47:04.889412 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.889367    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3"} err="failed to get container status \"67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3\": rpc error: code = NotFound desc = could not find container \"67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3\": container with ID starting with 67637a0067e25470857e2c092404f628a21378fed9f4c7ad98d9736eb8108cd3 not found: ID does not exist"
Sep 08 13:47:04.889412 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.889399    2561 scope.go:117] "RemoveContainer" containerID="da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7"
Sep 08 13:47:04.889690 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:47:04.889672    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7\": container with ID starting with da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7 not found: ID does not exist" containerID="da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7"
Sep 08 13:47:04.889731 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.889697    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7"} err="failed to get container status \"da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7\": rpc error: code = NotFound desc = could not find container \"da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7\": container with ID starting with da996ed9e2b6de998d46050395e8ffba4004c371b1f69379c0caf0c9344853a7 not found: ID does not exist"
Sep 08 13:47:04.893620 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.893591    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"]
Sep 08 13:47:04.897836 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.897809    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b719d2-openshift-default-65496b95cb-b8prg"]
Sep 08 13:47:04.959852 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.959757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97csm\" (UniqueName: \"kubernetes.io/projected/ecaedac2-3f93-49fa-bba4-d5a674894361-kube-api-access-97csm\") pod \"payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6\" (UID: \"ecaedac2-3f93-49fa-bba4-d5a674894361\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:04.959852 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.959806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaedac2-3f93-49fa-bba4-d5a674894361-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6\" (UID: \"ecaedac2-3f93-49fa-bba4-d5a674894361\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:04.960536 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.960516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaedac2-3f93-49fa-bba4-d5a674894361-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6\" (UID: \"ecaedac2-3f93-49fa-bba4-d5a674894361\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:04.968821 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:04.968783    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97csm\" (UniqueName: \"kubernetes.io/projected/ecaedac2-3f93-49fa-bba4-d5a674894361-kube-api-access-97csm\") pod \"payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6\" (UID: \"ecaedac2-3f93-49fa-bba4-d5a674894361\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:05.014929 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.014897    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:05.071753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.071726    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:05.292286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.292200    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"]
Sep 08 13:47:05.294724 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:47:05.294687    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ed378e_38c7_4f27_b79a_117601145a68.slice/crio-5eca1899cfac6cd775e8f73b9fe331bdbe4b023c82289bb61282d564487495ef WatchSource:0}: Error finding container 5eca1899cfac6cd775e8f73b9fe331bdbe4b023c82289bb61282d564487495ef: Status 404 returned error can't find the container with id 5eca1899cfac6cd775e8f73b9fe331bdbe4b023c82289bb61282d564487495ef
Sep 08 13:47:05.357030 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.357004    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"]
Sep 08 13:47:05.359140 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:47:05.359116    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecaedac2_3f93_49fa_bba4_d5a674894361.slice/crio-7f1d2ff716e35613ccd7db98d2b7a094d81b29bf3f4b36f2ab0c00872f414352 WatchSource:0}: Error finding container 7f1d2ff716e35613ccd7db98d2b7a094d81b29bf3f4b36f2ab0c00872f414352: Status 404 returned error can't find the container with id 7f1d2ff716e35613ccd7db98d2b7a094d81b29bf3f4b36f2ab0c00872f414352
Sep 08 13:47:05.878620 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.878584    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6" event={"ID":"ecaedac2-3f93-49fa-bba4-d5a674894361","Type":"ContainerStarted","Data":"338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4"}
Sep 08 13:47:05.878834 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.878646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6" event={"ID":"ecaedac2-3f93-49fa-bba4-d5a674894361","Type":"ContainerStarted","Data":"7f1d2ff716e35613ccd7db98d2b7a094d81b29bf3f4b36f2ab0c00872f414352"}
Sep 08 13:47:05.878834 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.878677    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:05.880254 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.880220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm" event={"ID":"59ed378e-38c7-4f27-b79a-117601145a68","Type":"ContainerStarted","Data":"f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80"}
Sep 08 13:47:05.880354 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.880272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm" event={"ID":"59ed378e-38c7-4f27-b79a-117601145a68","Type":"ContainerStarted","Data":"5eca1899cfac6cd775e8f73b9fe331bdbe4b023c82289bb61282d564487495ef"}
Sep 08 13:47:05.880354 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.880301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:05.896896 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.896828    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6" podStartSLOduration=1.8968132949999998 podStartE2EDuration="1.896813295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:05.895223877 +0000 UTC m=+1134.246209824" watchObservedRunningTime="2026-09-08 13:47:05.896813295 +0000 UTC m=+1134.247799243"
Sep 08 13:47:05.913900 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:05.913847    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm" podStartSLOduration=1.913829808 podStartE2EDuration="1.913829808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:05.912565809 +0000 UTC m=+1134.263551768" watchObservedRunningTime="2026-09-08 13:47:05.913829808 +0000 UTC m=+1134.264815756"
Sep 08 13:47:06.254263 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:06.254223    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c26bcd53-9ee2-4082-809b-b28852380d9a" path="/var/lib/kubelet/pods/c26bcd53-9ee2-4082-809b-b28852380d9a/volumes"
Sep 08 13:47:10.756326 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:10.756249    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m" podUID="ca4d15fa-1034-4bb1-b58b-a53819eaa407" containerName="main" containerID="cri-o://c1b3d87e654c6446d4d30bbc38cdafd934e1ed33ff5beedfdaa0f30cd1031ce8" gracePeriod=45
Sep 08 13:47:10.906900 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:10.906866    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca4d15fa-1034-4bb1-b58b-a53819eaa407" containerID="c1b3d87e654c6446d4d30bbc38cdafd934e1ed33ff5beedfdaa0f30cd1031ce8" exitCode=0
Sep 08 13:47:10.907078 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:10.906939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m" event={"ID":"ca4d15fa-1034-4bb1-b58b-a53819eaa407","Type":"ContainerDied","Data":"c1b3d87e654c6446d4d30bbc38cdafd934e1ed33ff5beedfdaa0f30cd1031ce8"}
Sep 08 13:47:11.013821 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.013763    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:47:11.117126 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tmp-dir\") pod \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") "
Sep 08 13:47:11.117338 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117150    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tls-certs\") pod \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") "
Sep 08 13:47:11.117338 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117174    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-model-cache\") pod \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") "
Sep 08 13:47:11.117338 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117200    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-dshm\") pod \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") "
Sep 08 13:47:11.117338 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbknj\" (UniqueName: \"kubernetes.io/projected/ca4d15fa-1034-4bb1-b58b-a53819eaa407-kube-api-access-fbknj\") pod \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") "
Sep 08 13:47:11.117338 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117275    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-home\") pod \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\" (UID: \"ca4d15fa-1034-4bb1-b58b-a53819eaa407\") "
Sep 08 13:47:11.117612 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117500    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca4d15fa-1034-4bb1-b58b-a53819eaa407" (UID: "ca4d15fa-1034-4bb1-b58b-a53819eaa407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:11.117612 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117515    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca4d15fa-1034-4bb1-b58b-a53819eaa407" (UID: "ca4d15fa-1034-4bb1-b58b-a53819eaa407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:11.117612 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.117565    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-home" (OuterVolumeSpecName: "home") pod "ca4d15fa-1034-4bb1-b58b-a53819eaa407" (UID: "ca4d15fa-1034-4bb1-b58b-a53819eaa407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:11.119391 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.119347    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-dshm" (OuterVolumeSpecName: "dshm") pod "ca4d15fa-1034-4bb1-b58b-a53819eaa407" (UID: "ca4d15fa-1034-4bb1-b58b-a53819eaa407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:11.119499 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.119405    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca4d15fa-1034-4bb1-b58b-a53819eaa407" (UID: "ca4d15fa-1034-4bb1-b58b-a53819eaa407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:11.119499 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.119414    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4d15fa-1034-4bb1-b58b-a53819eaa407-kube-api-access-fbknj" (OuterVolumeSpecName: "kube-api-access-fbknj") pod "ca4d15fa-1034-4bb1-b58b-a53819eaa407" (UID: "ca4d15fa-1034-4bb1-b58b-a53819eaa407"). InnerVolumeSpecName "kube-api-access-fbknj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:11.218147 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.218112    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbknj\" (UniqueName: \"kubernetes.io/projected/ca4d15fa-1034-4bb1-b58b-a53819eaa407-kube-api-access-fbknj\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:11.218147 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.218144    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:11.218147 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.218156    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:11.218410 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.218166    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4d15fa-1034-4bb1-b58b-a53819eaa407-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:11.218410 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.218173    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:11.218410 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.218180    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4d15fa-1034-4bb1-b58b-a53819eaa407-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:11.913655 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.913625    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"
Sep 08 13:47:11.914083 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.913658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m" event={"ID":"ca4d15fa-1034-4bb1-b58b-a53819eaa407","Type":"ContainerDied","Data":"2eaacd2787ece7b8fd755fca6db820af67dbf0d993ff1b6d77a2f6b334842457"}
Sep 08 13:47:11.914083 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.913712    2561 scope.go:117] "RemoveContainer" containerID="c1b3d87e654c6446d4d30bbc38cdafd934e1ed33ff5beedfdaa0f30cd1031ce8"
Sep 08 13:47:11.941612 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.941569    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"]
Sep 08 13:47:11.943850 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:11.943816    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-c2512ae8-kserve-5cb4cf9f88-q572m"]
Sep 08 13:47:12.261816 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:12.261776    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4d15fa-1034-4bb1-b58b-a53819eaa407" path="/var/lib/kubelet/pods/ca4d15fa-1034-4bb1-b58b-a53819eaa407/volumes"
Sep 08 13:47:16.888522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:16.888491    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:16.889086 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:16.888575    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:17.895549 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:17.895509    2561 patch_prober.go:28] interesting pod/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 13:47:17.896024 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:17.895578    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 13:47:18.953701 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:18.953663    2561 generic.go:356] "Generic (PLEG): container finished" podID="90523b1d-fc1f-4795-aacc-5335435256cf" containerID="e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1" exitCode=0
Sep 08 13:47:18.954132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:18.953737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" event={"ID":"90523b1d-fc1f-4795-aacc-5335435256cf","Type":"ContainerDied","Data":"e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1"}
Sep 08 13:47:18.954132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:18.954128    2561 scope.go:117] "RemoveContainer" containerID="e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1"
Sep 08 13:47:18.955175 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:18.955151    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:47:18.955234 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:18.955217    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:47:18.955274 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:18.955247    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:47:19.962992 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:19.962951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" event={"ID":"90523b1d-fc1f-4795-aacc-5335435256cf","Type":"ContainerStarted","Data":"e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a"}
Sep 08 13:47:20.166970 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:20.166930    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:20.167245 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:20.167221    2561 patch_prober.go:28] interesting pod/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 13:47:20.167294 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:20.167274    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 13:47:21.166654 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:21.166618    2561 patch_prober.go:28] interesting pod/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 13:47:21.167172 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:21.166679    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 13:47:22.166570 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:22.166534    2561 patch_prober.go:28] interesting pod/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 13:47:22.166760 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:22.166605    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 13:47:23.166996 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:23.166955    2561 patch_prober.go:28] interesting pod/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 13:47:23.167435 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:23.167019    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 13:47:24.166830 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:24.166792    2561 patch_prober.go:28] interesting pod/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 13:47:24.167044 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:24.166855    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 13:47:25.280899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:25.280861    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:25.281420 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:25.281397    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:25.282616 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:25.282596    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:47:27.037831 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.037796    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"]
Sep 08 13:47:27.038199 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.038113    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4d15fa-1034-4bb1-b58b-a53819eaa407" containerName="main"
Sep 08 13:47:27.038199 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.038124    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4d15fa-1034-4bb1-b58b-a53819eaa407" containerName="main"
Sep 08 13:47:27.038199 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.038172    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4d15fa-1034-4bb1-b58b-a53819eaa407" containerName="main"
Sep 08 13:47:27.040724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.040698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.043554 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.043529    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a7ef4b\"/\"default-dockercfg-z9l4v\""
Sep 08 13:47:27.044816 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.044791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a7ef4b\"/\"kube-root-ca.crt\""
Sep 08 13:47:27.044961 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.044891    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a7ef4b\"/\"test-model-a7ef4b-kserve-self-signed-certs\""
Sep 08 13:47:27.044961 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.044903    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a7ef4b\"/\"openshift-service-ca.crt\""
Sep 08 13:47:27.051947 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.051925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"]
Sep 08 13:47:27.161395 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.161352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-dshm\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.161595 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.161454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm9rm\" (UniqueName: \"kubernetes.io/projected/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-kube-api-access-cm9rm\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.161595 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.161516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-model-cache\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.161595 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.161543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tls-certs\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.161744 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.161660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tmp-dir\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.161744 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.161715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-home\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.262786 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.262745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tmp-dir\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.262786 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.262791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-home\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.263045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.262820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-dshm\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.263045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.262840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm9rm\" (UniqueName: \"kubernetes.io/projected/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-kube-api-access-cm9rm\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.263045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.262955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-model-cache\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.263045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.263004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tls-certs\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.263223 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.263194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-home\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.263293 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.263274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tmp-dir\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.263293 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.263285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-model-cache\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.265081 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.265058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-dshm\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.265343 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.265326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tls-certs\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.270573 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.270551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm9rm\" (UniqueName: \"kubernetes.io/projected/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-kube-api-access-cm9rm\") pod \"test-model-a7ef4b-kserve-6cdb576d99-8m7jj\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") " pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.352002 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.351914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:27.626449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:27.626421    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"]
Sep 08 13:47:27.628095 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:47:27.628057    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e6fff33_996e_4f6b_ace3_c3e82fb18beb.slice/crio-0330e206709d79001bf80cbc74173bc0a972ffd4076e28aec4a65a8fe0c48576 WatchSource:0}: Error finding container 0330e206709d79001bf80cbc74173bc0a972ffd4076e28aec4a65a8fe0c48576: Status 404 returned error can't find the container with id 0330e206709d79001bf80cbc74173bc0a972ffd4076e28aec4a65a8fe0c48576
Sep 08 13:47:28.002838 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:28.002796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj" event={"ID":"1e6fff33-996e-4f6b-ace3-c3e82fb18beb","Type":"ContainerStarted","Data":"d0194dde51be24a3862af1eff2434ffbee2317708ff445248c53fd6bf4143115"}
Sep 08 13:47:28.002838 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:28.002844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj" event={"ID":"1e6fff33-996e-4f6b-ace3-c3e82fb18beb","Type":"ContainerStarted","Data":"0330e206709d79001bf80cbc74173bc0a972ffd4076e28aec4a65a8fe0c48576"}
Sep 08 13:47:28.020181 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:28.020126    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj" podStartSLOduration=1.020112909 podStartE2EDuration="1.020112909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:28.019850336 +0000 UTC m=+1156.370836284" watchObservedRunningTime="2026-09-08 13:47:28.020112909 +0000 UTC m=+1156.371098856"
Sep 08 13:47:36.235286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.235207    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"]
Sep 08 13:47:36.238713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.238695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.241473 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.241443    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-d60b5a\"/\"kube-root-ca.crt\""
Sep 08 13:47:36.241655 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.241519    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-d60b5a\"/\"openshift-service-ca.crt\""
Sep 08 13:47:36.241655 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.241560    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-d60b5a\"/\"default-dockercfg-7b9ps\""
Sep 08 13:47:36.241939 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.241917    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-d60b5a\"/\"explicit-model-d60b5a-kserve-self-signed-certs\""
Sep 08 13:47:36.244811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.244781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-model-cache\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.244931 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.244834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-dshm\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.245002 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.244934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5lgv\" (UniqueName: \"kubernetes.io/projected/029394e9-d753-43b8-9ba1-bf5823d6b895-kube-api-access-x5lgv\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.245002 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.244989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-home\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.245169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.245150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-tmp-dir\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.245221 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.245201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029394e9-d753-43b8-9ba1-bf5823d6b895-tls-certs\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.251776 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.251752    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"]
Sep 08 13:47:36.345822 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.345787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-model-cache\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.345822 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.345823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-dshm\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.346074 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.345853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5lgv\" (UniqueName: \"kubernetes.io/projected/029394e9-d753-43b8-9ba1-bf5823d6b895-kube-api-access-x5lgv\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.346074 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.345881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-home\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.346074 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.345936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-tmp-dir\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.346074 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.346055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029394e9-d753-43b8-9ba1-bf5823d6b895-tls-certs\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.346335 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.346214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-model-cache\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.346335 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.346252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-tmp-dir\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.346335 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.346320    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-home\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.348115 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.348086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-dshm\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.348350 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.348334    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029394e9-d753-43b8-9ba1-bf5823d6b895-tls-certs\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.357740 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.357711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5lgv\" (UniqueName: \"kubernetes.io/projected/029394e9-d753-43b8-9ba1-bf5823d6b895-kube-api-access-x5lgv\") pod \"explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") " pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.549678 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.549564    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:36.821428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:36.821402    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"]
Sep 08 13:47:36.823475 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:47:36.823447    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod029394e9_d753_43b8_9ba1_bf5823d6b895.slice/crio-2a52da50ad160e56132c8b5371a1ce11cd291b12ef359c88233979dfaf18b2dc WatchSource:0}: Error finding container 2a52da50ad160e56132c8b5371a1ce11cd291b12ef359c88233979dfaf18b2dc: Status 404 returned error can't find the container with id 2a52da50ad160e56132c8b5371a1ce11cd291b12ef359c88233979dfaf18b2dc
Sep 08 13:47:37.047615 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:37.047575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp" event={"ID":"029394e9-d753-43b8-9ba1-bf5823d6b895","Type":"ContainerStarted","Data":"70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75"}
Sep 08 13:47:37.047615 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:37.047627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp" event={"ID":"029394e9-d753-43b8-9ba1-bf5823d6b895","Type":"ContainerStarted","Data":"2a52da50ad160e56132c8b5371a1ce11cd291b12ef359c88233979dfaf18b2dc"}
Sep 08 13:47:37.066772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:37.066701    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp" podStartSLOduration=1.066677159 podStartE2EDuration="1.066677159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:37.065702474 +0000 UTC m=+1165.416688423" watchObservedRunningTime="2026-09-08 13:47:37.066677159 +0000 UTC m=+1165.417663107"
Sep 08 13:47:37.352449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:37.352417    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:37.352856 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:37.352669    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:37.365427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:37.365399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:38.064301 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:38.064268    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:47:45.658164 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:45.658127    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"]
Sep 08 13:47:46.550403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:46.550359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:47:56.598961 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.598925    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"]
Sep 08 13:47:56.599439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.599246    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm" podUID="59ed378e-38c7-4f27-b79a-117601145a68" containerName="payload-processing" containerID="cri-o://f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80" gracePeriod=30
Sep 08 13:47:56.607980 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.607952    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"]
Sep 08 13:47:56.608216 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.608193    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6" podUID="ecaedac2-3f93-49fa-bba4-d5a674894361" containerName="payload-pre-processing" containerID="cri-o://338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4" gracePeriod=30
Sep 08 13:47:56.849087 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.849028    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:56.880295 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.880271    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:56.916567 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.916528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/59ed378e-38c7-4f27-b79a-117601145a68-kube-api-access-bwgkm\") pod \"59ed378e-38c7-4f27-b79a-117601145a68\" (UID: \"59ed378e-38c7-4f27-b79a-117601145a68\") "
Sep 08 13:47:56.917713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.916609    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97csm\" (UniqueName: \"kubernetes.io/projected/ecaedac2-3f93-49fa-bba4-d5a674894361-kube-api-access-97csm\") pod \"ecaedac2-3f93-49fa-bba4-d5a674894361\" (UID: \"ecaedac2-3f93-49fa-bba4-d5a674894361\") "
Sep 08 13:47:56.917713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.916696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaedac2-3f93-49fa-bba4-d5a674894361-plugins-config-volume\") pod \"ecaedac2-3f93-49fa-bba4-d5a674894361\" (UID: \"ecaedac2-3f93-49fa-bba4-d5a674894361\") "
Sep 08 13:47:56.917713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.916765    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ed378e-38c7-4f27-b79a-117601145a68-plugins-config-volume\") pod \"59ed378e-38c7-4f27-b79a-117601145a68\" (UID: \"59ed378e-38c7-4f27-b79a-117601145a68\") "
Sep 08 13:47:56.917984 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.917942    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaedac2-3f93-49fa-bba4-d5a674894361-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecaedac2-3f93-49fa-bba4-d5a674894361" (UID: "ecaedac2-3f93-49fa-bba4-d5a674894361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:56.918096 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.918064    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ed378e-38c7-4f27-b79a-117601145a68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59ed378e-38c7-4f27-b79a-117601145a68" (UID: "59ed378e-38c7-4f27-b79a-117601145a68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:56.919753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.919728    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaedac2-3f93-49fa-bba4-d5a674894361-kube-api-access-97csm" (OuterVolumeSpecName: "kube-api-access-97csm") pod "ecaedac2-3f93-49fa-bba4-d5a674894361" (UID: "ecaedac2-3f93-49fa-bba4-d5a674894361"). InnerVolumeSpecName "kube-api-access-97csm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:56.919864 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:56.919760    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ed378e-38c7-4f27-b79a-117601145a68-kube-api-access-bwgkm" (OuterVolumeSpecName: "kube-api-access-bwgkm") pod "59ed378e-38c7-4f27-b79a-117601145a68" (UID: "59ed378e-38c7-4f27-b79a-117601145a68"). InnerVolumeSpecName "kube-api-access-bwgkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:57.017618 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.017586    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97csm\" (UniqueName: \"kubernetes.io/projected/ecaedac2-3f93-49fa-bba4-d5a674894361-kube-api-access-97csm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:57.017618 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.017615    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaedac2-3f93-49fa-bba4-d5a674894361-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:57.017618 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.017627    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ed378e-38c7-4f27-b79a-117601145a68-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:57.017846 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.017638    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwgkm\" (UniqueName: \"kubernetes.io/projected/59ed378e-38c7-4f27-b79a-117601145a68-kube-api-access-bwgkm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:47:57.143289 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.143197    2561 generic.go:356] "Generic (PLEG): container finished" podID="ecaedac2-3f93-49fa-bba4-d5a674894361" containerID="338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4" exitCode=0
Sep 08 13:47:57.143289 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.143246    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"
Sep 08 13:47:57.143289 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.143270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6" event={"ID":"ecaedac2-3f93-49fa-bba4-d5a674894361","Type":"ContainerDied","Data":"338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4"}
Sep 08 13:47:57.143607 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.143326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6" event={"ID":"ecaedac2-3f93-49fa-bba4-d5a674894361","Type":"ContainerDied","Data":"7f1d2ff716e35613ccd7db98d2b7a094d81b29bf3f4b36f2ab0c00872f414352"}
Sep 08 13:47:57.143607 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.143352    2561 scope.go:117] "RemoveContainer" containerID="338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4"
Sep 08 13:47:57.144638 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.144616    2561 generic.go:356] "Generic (PLEG): container finished" podID="59ed378e-38c7-4f27-b79a-117601145a68" containerID="f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80" exitCode=0
Sep 08 13:47:57.144737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.144654    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"
Sep 08 13:47:57.144737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.144707    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm" event={"ID":"59ed378e-38c7-4f27-b79a-117601145a68","Type":"ContainerDied","Data":"f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80"}
Sep 08 13:47:57.144828 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.144774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm" event={"ID":"59ed378e-38c7-4f27-b79a-117601145a68","Type":"ContainerDied","Data":"5eca1899cfac6cd775e8f73b9fe331bdbe4b023c82289bb61282d564487495ef"}
Sep 08 13:47:57.153523 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.153500    2561 scope.go:117] "RemoveContainer" containerID="338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4"
Sep 08 13:47:57.153815 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:47:57.153796    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4\": container with ID starting with 338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4 not found: ID does not exist" containerID="338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4"
Sep 08 13:47:57.153900 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.153826    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4"} err="failed to get container status \"338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4\": rpc error: code = NotFound desc = could not find container \"338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4\": container with ID starting with 338f1826a75e44ca5e39679f5cf5e4f78042b4c57b44f745644ba5b3dd20c6a4 not found: ID does not exist"
Sep 08 13:47:57.153900 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.153846    2561 scope.go:117] "RemoveContainer" containerID="f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80"
Sep 08 13:47:57.161876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.161858    2561 scope.go:117] "RemoveContainer" containerID="f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80"
Sep 08 13:47:57.162130 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:47:57.162111    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80\": container with ID starting with f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80 not found: ID does not exist" containerID="f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80"
Sep 08 13:47:57.162192 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.162141    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80"} err="failed to get container status \"f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80\": rpc error: code = NotFound desc = could not find container \"f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80\": container with ID starting with f827f89709d0f6d61021b08974c899467ecab28fc365f2015e9f9bd57f362d80 not found: ID does not exist"
Sep 08 13:47:57.166686 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.166663    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"]
Sep 08 13:47:57.170706 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.170682    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d60b5a-f54c8cc5d-fdph6"]
Sep 08 13:47:57.181556 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.181532    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"]
Sep 08 13:47:57.185535 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:57.185511    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d60b5a-7c55678685-xpvpm"]
Sep 08 13:47:58.245363 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:58.245329    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ed378e-38c7-4f27-b79a-117601145a68" path="/var/lib/kubelet/pods/59ed378e-38c7-4f27-b79a-117601145a68/volumes"
Sep 08 13:47:58.245732 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:47:58.245703    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecaedac2-3f93-49fa-bba4-d5a674894361" path="/var/lib/kubelet/pods/ecaedac2-3f93-49fa-bba4-d5a674894361/volumes"
Sep 08 13:48:00.678506 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:00.678461    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp" podUID="029394e9-d753-43b8-9ba1-bf5823d6b895" containerName="main" containerID="cri-o://70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75" gracePeriod=45
Sep 08 13:48:00.925043 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:00.925014    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:48:01.056229 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-model-cache\") pod \"029394e9-d753-43b8-9ba1-bf5823d6b895\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") "
Sep 08 13:48:01.056229 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-dshm\") pod \"029394e9-d753-43b8-9ba1-bf5823d6b895\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") "
Sep 08 13:48:01.056229 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056210    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029394e9-d753-43b8-9ba1-bf5823d6b895-tls-certs\") pod \"029394e9-d753-43b8-9ba1-bf5823d6b895\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") "
Sep 08 13:48:01.056550 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056276    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-home\") pod \"029394e9-d753-43b8-9ba1-bf5823d6b895\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") "
Sep 08 13:48:01.056550 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056303    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-tmp-dir\") pod \"029394e9-d753-43b8-9ba1-bf5823d6b895\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") "
Sep 08 13:48:01.056550 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056329    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5lgv\" (UniqueName: \"kubernetes.io/projected/029394e9-d753-43b8-9ba1-bf5823d6b895-kube-api-access-x5lgv\") pod \"029394e9-d753-43b8-9ba1-bf5823d6b895\" (UID: \"029394e9-d753-43b8-9ba1-bf5823d6b895\") "
Sep 08 13:48:01.056550 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056466    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-model-cache" (OuterVolumeSpecName: "model-cache") pod "029394e9-d753-43b8-9ba1-bf5823d6b895" (UID: "029394e9-d753-43b8-9ba1-bf5823d6b895"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:01.056734 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056628    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-home" (OuterVolumeSpecName: "home") pod "029394e9-d753-43b8-9ba1-bf5823d6b895" (UID: "029394e9-d753-43b8-9ba1-bf5823d6b895"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:01.056734 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.056638    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "029394e9-d753-43b8-9ba1-bf5823d6b895" (UID: "029394e9-d753-43b8-9ba1-bf5823d6b895"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:01.058619 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.058590    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-dshm" (OuterVolumeSpecName: "dshm") pod "029394e9-d753-43b8-9ba1-bf5823d6b895" (UID: "029394e9-d753-43b8-9ba1-bf5823d6b895"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:01.058750 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.058625    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029394e9-d753-43b8-9ba1-bf5823d6b895-kube-api-access-x5lgv" (OuterVolumeSpecName: "kube-api-access-x5lgv") pod "029394e9-d753-43b8-9ba1-bf5823d6b895" (UID: "029394e9-d753-43b8-9ba1-bf5823d6b895"). InnerVolumeSpecName "kube-api-access-x5lgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:01.058750 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.058688    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029394e9-d753-43b8-9ba1-bf5823d6b895-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "029394e9-d753-43b8-9ba1-bf5823d6b895" (UID: "029394e9-d753-43b8-9ba1-bf5823d6b895"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:48:01.157778 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.157741    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:01.157778 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.157771    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:01.157778 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.157782    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5lgv\" (UniqueName: \"kubernetes.io/projected/029394e9-d753-43b8-9ba1-bf5823d6b895-kube-api-access-x5lgv\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:01.157778 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.157791    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:01.158051 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.157800    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029394e9-d753-43b8-9ba1-bf5823d6b895-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:01.158051 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.157808    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029394e9-d753-43b8-9ba1-bf5823d6b895-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:01.167114 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.167086    2561 generic.go:356] "Generic (PLEG): container finished" podID="029394e9-d753-43b8-9ba1-bf5823d6b895" containerID="70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75" exitCode=0
Sep 08 13:48:01.167257 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.167139    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"
Sep 08 13:48:01.167257 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.167176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp" event={"ID":"029394e9-d753-43b8-9ba1-bf5823d6b895","Type":"ContainerDied","Data":"70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75"}
Sep 08 13:48:01.167257 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.167224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp" event={"ID":"029394e9-d753-43b8-9ba1-bf5823d6b895","Type":"ContainerDied","Data":"2a52da50ad160e56132c8b5371a1ce11cd291b12ef359c88233979dfaf18b2dc"}
Sep 08 13:48:01.167257 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.167249    2561 scope.go:117] "RemoveContainer" containerID="70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75"
Sep 08 13:48:01.176509 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.176492    2561 scope.go:117] "RemoveContainer" containerID="70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75"
Sep 08 13:48:01.176770 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:48:01.176750    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75\": container with ID starting with 70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75 not found: ID does not exist" containerID="70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75"
Sep 08 13:48:01.176815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.176778    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75"} err="failed to get container status \"70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75\": rpc error: code = NotFound desc = could not find container \"70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75\": container with ID starting with 70c0ef18bf47d01e2336d53e4eba2a3ec1de50ec67534723b193ef916eedff75 not found: ID does not exist"
Sep 08 13:48:01.189065 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.189040    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"]
Sep 08 13:48:01.193121 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:01.193099    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-d60b5a/explicit-model-d60b5a-kserve-fcd8f96f7-6hrpp"]
Sep 08 13:48:02.245051 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:02.245017    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="029394e9-d753-43b8-9ba1-bf5823d6b895" path="/var/lib/kubelet/pods/029394e9-d753-43b8-9ba1-bf5823d6b895/volumes"
Sep 08 13:48:03.620277 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:03.620241    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"]
Sep 08 13:48:03.620720 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:03.620514    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy" containerID="cri-o://e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a" gracePeriod=30
Sep 08 13:48:04.468940 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.468908    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"]
Sep 08 13:48:04.469211 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469199    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ed378e-38c7-4f27-b79a-117601145a68" containerName="payload-processing"
Sep 08 13:48:04.469256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469214    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ed378e-38c7-4f27-b79a-117601145a68" containerName="payload-processing"
Sep 08 13:48:04.469256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469231    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecaedac2-3f93-49fa-bba4-d5a674894361" containerName="payload-pre-processing"
Sep 08 13:48:04.469256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469236    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecaedac2-3f93-49fa-bba4-d5a674894361" containerName="payload-pre-processing"
Sep 08 13:48:04.469256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469244    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029394e9-d753-43b8-9ba1-bf5823d6b895" containerName="main"
Sep 08 13:48:04.469256 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469249    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="029394e9-d753-43b8-9ba1-bf5823d6b895" containerName="main"
Sep 08 13:48:04.469427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469301    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="029394e9-d753-43b8-9ba1-bf5823d6b895" containerName="main"
Sep 08 13:48:04.469427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469310    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ed378e-38c7-4f27-b79a-117601145a68" containerName="payload-processing"
Sep 08 13:48:04.469427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.469317    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecaedac2-3f93-49fa-bba4-d5a674894361" containerName="payload-pre-processing"
Sep 08 13:48:04.473471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.473449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.476309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.476283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-dca93a-openshift-default-dockercfg-4r5qc\""
Sep 08 13:48:04.486510 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.486488    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"]
Sep 08 13:48:04.581184 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-data\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.581184 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-envoy\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.581472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-token\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.581472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsfv\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-kube-api-access-dhsfv\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.581472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8822de4-f6ef-4a30-b718-8d820d49ffef-istiod-ca-cert\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.581472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-credential-socket\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.581472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-certs\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.581728 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-podinfo\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.581728 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.581541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-socket\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682236 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-envoy\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682236 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-token\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsfv\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-kube-api-access-dhsfv\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682292    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8822de4-f6ef-4a30-b718-8d820d49ffef-istiod-ca-cert\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-credential-socket\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-certs\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-podinfo\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-socket\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.682810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-data\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.683152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682836    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-certs\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.683152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-credential-socket\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.683152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.682947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-socket\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.683152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.683012    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-data\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.683315 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.683152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8822de4-f6ef-4a30-b718-8d820d49ffef-istiod-ca-cert\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.684685 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.684660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-envoy\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.685133 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.685115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-podinfo\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.706903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.706877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsfv\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-kube-api-access-dhsfv\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.708061 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.708031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-token\") pod \"e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") " pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:04.783469 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:04.783341    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:05.057670 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:05.057553    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"]
Sep 08 13:48:05.061652 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:48:05.061618    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8822de4_f6ef_4a30_b718_8d820d49ffef.slice/crio-e19148e7032320ddd59aa613842fc4310f2bf1302534f37e99bbe97f6e3c1dc8 WatchSource:0}: Error finding container e19148e7032320ddd59aa613842fc4310f2bf1302534f37e99bbe97f6e3c1dc8: Status 404 returned error can't find the container with id e19148e7032320ddd59aa613842fc4310f2bf1302534f37e99bbe97f6e3c1dc8
Sep 08 13:48:05.063654 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:05.063620    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:05.063737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:05.063685    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:05.063737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:05.063714    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:05.188600 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:05.188564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8" event={"ID":"e8822de4-f6ef-4a30-b718-8d820d49ffef","Type":"ContainerStarted","Data":"368dd458bde5dbd84dbdd51efe96a39a0da4dd1ec40a596b8c016b1f68ae86e2"}
Sep 08 13:48:05.188751 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:05.188621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8" event={"ID":"e8822de4-f6ef-4a30-b718-8d820d49ffef","Type":"ContainerStarted","Data":"e19148e7032320ddd59aa613842fc4310f2bf1302534f37e99bbe97f6e3c1dc8"}
Sep 08 13:48:05.215288 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:05.215224    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8" podStartSLOduration=1.215202174 podStartE2EDuration="1.215202174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:05.212127507 +0000 UTC m=+1193.563113456" watchObservedRunningTime="2026-09-08 13:48:05.215202174 +0000 UTC m=+1193.566188125"
Sep 08 13:48:05.784247 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:05.784212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:06.788529 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:06.788496    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:07.199323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:07.199293    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:07.200282 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:07.200259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:08.869461 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:08.869436    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:48:09.018581 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018542    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-credential-socket\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.018772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018610    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90523b1d-fc1f-4795-aacc-5335435256cf-istio-podinfo\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.018772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-certs\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.018772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018645    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-envoy\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.018772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018673    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-istio-token\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.019007 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018765    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-data\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.019007 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018823    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-socket\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.019007 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90523b1d-fc1f-4795-aacc-5335435256cf-istiod-ca-cert\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.019007 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018851    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:09.019007 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018943    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbw6w\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-kube-api-access-lbw6w\") pod \"90523b1d-fc1f-4795-aacc-5335435256cf\" (UID: \"90523b1d-fc1f-4795-aacc-5335435256cf\") "
Sep 08 13:48:09.019007 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.018980    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:09.019323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.019212    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:09.019323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.019241    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.019323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.019261    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.019323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.019265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:09.019323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.019307    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90523b1d-fc1f-4795-aacc-5335435256cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:48:09.020954 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.020924    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:09.020954 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.020937    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:09.021108 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.020950    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90523b1d-fc1f-4795-aacc-5335435256cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:48:09.021207 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.021189    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-kube-api-access-lbw6w" (OuterVolumeSpecName: "kube-api-access-lbw6w") pod "90523b1d-fc1f-4795-aacc-5335435256cf" (UID: "90523b1d-fc1f-4795-aacc-5335435256cf"). InnerVolumeSpecName "kube-api-access-lbw6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:09.120417 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.120357    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90523b1d-fc1f-4795-aacc-5335435256cf-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.120417 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.120414    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.120417 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.120424    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.120417 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.120434    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.120693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.120442    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90523b1d-fc1f-4795-aacc-5335435256cf-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.120693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.120451    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90523b1d-fc1f-4795-aacc-5335435256cf-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.120693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.120461    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbw6w\" (UniqueName: \"kubernetes.io/projected/90523b1d-fc1f-4795-aacc-5335435256cf-kube-api-access-lbw6w\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:09.209777 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.209744    2561 generic.go:356] "Generic (PLEG): container finished" podID="90523b1d-fc1f-4795-aacc-5335435256cf" containerID="e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a" exitCode=0
Sep 08 13:48:09.209933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.209790    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"
Sep 08 13:48:09.209933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.209818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" event={"ID":"90523b1d-fc1f-4795-aacc-5335435256cf","Type":"ContainerDied","Data":"e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a"}
Sep 08 13:48:09.209933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.209867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f" event={"ID":"90523b1d-fc1f-4795-aacc-5335435256cf","Type":"ContainerDied","Data":"6a7e1ccacd56e3877f5334d94b7f5ee9153341cf9d54811ffc4f24ede3fc6cac"}
Sep 08 13:48:09.209933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.209889    2561 scope.go:117] "RemoveContainer" containerID="e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a"
Sep 08 13:48:09.219487 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.219470    2561 scope.go:117] "RemoveContainer" containerID="e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1"
Sep 08 13:48:09.227272 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.227255    2561 scope.go:117] "RemoveContainer" containerID="e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a"
Sep 08 13:48:09.227570 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:48:09.227552    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a\": container with ID starting with e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a not found: ID does not exist" containerID="e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a"
Sep 08 13:48:09.227624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.227578    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a"} err="failed to get container status \"e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a\": rpc error: code = NotFound desc = could not find container \"e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a\": container with ID starting with e2ea77140b5aa80f7576665ae48a6ede4a4b3c741c8a63d5e36410f89fd0be7a not found: ID does not exist"
Sep 08 13:48:09.227624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.227595    2561 scope.go:117] "RemoveContainer" containerID="e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1"
Sep 08 13:48:09.227848 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:48:09.227829    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1\": container with ID starting with e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1 not found: ID does not exist" containerID="e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1"
Sep 08 13:48:09.227890 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.227854    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1"} err="failed to get container status \"e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1\": rpc error: code = NotFound desc = could not find container \"e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1\": container with ID starting with e9e4e370779215159019241dd9a0e2ccfe4a2ba2203a54aef7571294d47743b1 not found: ID does not exist"
Sep 08 13:48:09.232169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.232149    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"]
Sep 08 13:48:09.239990 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:09.237792    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-d60b5a-openshift-default-6ddb6898d-rhk8f"]
Sep 08 13:48:10.244956 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:10.244921    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" path="/var/lib/kubelet/pods/90523b1d-fc1f-4795-aacc-5335435256cf/volumes"
Sep 08 13:48:15.061689 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.061648    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"]
Sep 08 13:48:15.062060 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.061928    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy"
Sep 08 13:48:15.062060 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.061939    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy"
Sep 08 13:48:15.062060 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.062006    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy"
Sep 08 13:48:15.066369 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.066349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.070328 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.070307    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-dca93a\"/\"default-dockercfg-tsvlw\""
Sep 08 13:48:15.070482 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.070338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-dca93a\"/\"kube-root-ca.crt\""
Sep 08 13:48:15.070482 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.070313    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-dca93a\"/\"openshift-service-ca.crt\""
Sep 08 13:48:15.070482 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.070371    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-dca93a\"/\"noten-model-dca93a-kserve-self-signed-certs\""
Sep 08 13:48:15.082769 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.082748    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"]
Sep 08 13:48:15.176090 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.176051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-dshm\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.176090 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.176088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-tmp-dir\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.176323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.176123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-model-cache\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.176323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.176231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-home\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.176323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.176264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c103696-11ed-4af1-aa56-d98845630485-tls-certs\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.176323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.176289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5brv5\" (UniqueName: \"kubernetes.io/projected/8c103696-11ed-4af1-aa56-d98845630485-kube-api-access-5brv5\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277105 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5brv5\" (UniqueName: \"kubernetes.io/projected/8c103696-11ed-4af1-aa56-d98845630485-kube-api-access-5brv5\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-dshm\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-tmp-dir\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-model-cache\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-home\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c103696-11ed-4af1-aa56-d98845630485-tls-certs\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277682 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-tmp-dir\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277682 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-model-cache\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.277766 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.277741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-home\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.279546 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.279517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-dshm\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.279823 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.279805    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c103696-11ed-4af1-aa56-d98845630485-tls-certs\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.286148 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.286126    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5brv5\" (UniqueName: \"kubernetes.io/projected/8c103696-11ed-4af1-aa56-d98845630485-kube-api-access-5brv5\") pod \"noten-model-dca93a-kserve-6c8bcb55bc-9rw8f\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") " pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.374841 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.374755    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:15.643075 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:15.643042    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"]
Sep 08 13:48:15.645300 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:48:15.645262    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c103696_11ed_4af1_aa56_d98845630485.slice/crio-aec5d1edfd50e9472ac734e849835fc9d269402f79b17d8841578bd8de70f816 WatchSource:0}: Error finding container aec5d1edfd50e9472ac734e849835fc9d269402f79b17d8841578bd8de70f816: Status 404 returned error can't find the container with id aec5d1edfd50e9472ac734e849835fc9d269402f79b17d8841578bd8de70f816
Sep 08 13:48:16.245172 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:16.245131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f" event={"ID":"8c103696-11ed-4af1-aa56-d98845630485","Type":"ContainerStarted","Data":"26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310"}
Sep 08 13:48:16.245172 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:16.245183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f" event={"ID":"8c103696-11ed-4af1-aa56-d98845630485","Type":"ContainerStarted","Data":"aec5d1edfd50e9472ac734e849835fc9d269402f79b17d8841578bd8de70f816"}
Sep 08 13:48:16.284662 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:16.284606    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f" podStartSLOduration=1.284590702 podStartE2EDuration="1.284590702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:16.282031479 +0000 UTC m=+1204.633017427" watchObservedRunningTime="2026-09-08 13:48:16.284590702 +0000 UTC m=+1204.635576669"
Sep 08 13:48:25.375533 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:25.375486    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:25.375533 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:25.375546    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:25.388461 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:25.388428    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:26.300985 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:26.300952    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:48:29.503001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.502959    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"]
Sep 08 13:48:29.503372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.503250    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy"
Sep 08 13:48:29.503372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.503262    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy"
Sep 08 13:48:29.503372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.503322    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90523b1d-fc1f-4795-aacc-5335435256cf" containerName="istio-proxy"
Sep 08 13:48:29.505729 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.505709    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.509032 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.509010    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f8154283-openshift-default-dockercfg-djsf6\""
Sep 08 13:48:29.524782 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.524755    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"]
Sep 08 13:48:29.601157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-credential-socket\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.601334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-socket\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.601334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgsj9\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-kube-api-access-qgsj9\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.601334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-certs\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.601334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-token\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.601334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-data\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.601334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-podinfo\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.601548 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-envoy\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.601548 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.601368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afef1e25-eb30-4967-973f-bcbd04a7ab61-istiod-ca-cert\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.701939 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.701893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-credential-socket\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.701955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-socket\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.701975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgsj9\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-kube-api-access-qgsj9\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.701993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-certs\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-token\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-data\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-podinfo\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-envoy\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afef1e25-eb30-4967-973f-bcbd04a7ab61-istiod-ca-cert\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-credential-socket\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-certs\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702729 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-data\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702729 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-socket\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.702964 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.702945    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afef1e25-eb30-4967-973f-bcbd04a7ab61-istiod-ca-cert\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.704513 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.704484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-envoy\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.704739 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.704718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-podinfo\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.711071 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.711050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgsj9\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-kube-api-access-qgsj9\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.711172 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.711056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-token\") pod \"e2e-mt-f8154283-openshift-default-687c55685b-48cbg\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") " pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:29.816651 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:29.816555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:30.095056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:30.095036    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"]
Sep 08 13:48:30.097291 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:48:30.097253    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafef1e25_eb30_4967_973f_bcbd04a7ab61.slice/crio-b07500cba2fbc0e636b38c00ee9580aa13dc36756a31461ec102de8f3910f6aa WatchSource:0}: Error finding container b07500cba2fbc0e636b38c00ee9580aa13dc36756a31461ec102de8f3910f6aa: Status 404 returned error can't find the container with id b07500cba2fbc0e636b38c00ee9580aa13dc36756a31461ec102de8f3910f6aa
Sep 08 13:48:30.099745 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:30.099713    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:30.099804 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:30.099789    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:30.099846 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:30.099817    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:30.310804 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:30.310765    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" event={"ID":"afef1e25-eb30-4967-973f-bcbd04a7ab61","Type":"ContainerStarted","Data":"e6b1016fa2cc756f7415938eedf212f4ac60d951991d507781fada38b5a292d9"}
Sep 08 13:48:30.311005 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:30.310825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" event={"ID":"afef1e25-eb30-4967-973f-bcbd04a7ab61","Type":"ContainerStarted","Data":"b07500cba2fbc0e636b38c00ee9580aa13dc36756a31461ec102de8f3910f6aa"}
Sep 08 13:48:30.336728 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:30.336670    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" podStartSLOduration=1.336654424 podStartE2EDuration="1.336654424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:30.335493939 +0000 UTC m=+1218.686479889" watchObservedRunningTime="2026-09-08 13:48:30.336654424 +0000 UTC m=+1218.687640373"
Sep 08 13:48:30.817328 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:30.817280    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:31.822468 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:31.822435    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:32.321091 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:32.321044    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:32.322027 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:32.322007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:34.528500 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.528461    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"]
Sep 08 13:48:34.531769 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.531747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:34.534355 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.534319    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f8154283\""
Sep 08 13:48:34.534566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.534423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8154283-dockercfg-6rbzg\""
Sep 08 13:48:34.542442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.542370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9fa65c-d3ba-4290-8f9d-c22baac63554-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh\" (UID: \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\") " pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:34.542593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.542466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxgh\" (UniqueName: \"kubernetes.io/projected/cc9fa65c-d3ba-4290-8f9d-c22baac63554-kube-api-access-pwxgh\") pod \"payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh\" (UID: \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\") " pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:34.543179 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.543156    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"]
Sep 08 13:48:34.643844 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.643794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9fa65c-d3ba-4290-8f9d-c22baac63554-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh\" (UID: \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\") " pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:34.644050 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.643853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxgh\" (UniqueName: \"kubernetes.io/projected/cc9fa65c-d3ba-4290-8f9d-c22baac63554-kube-api-access-pwxgh\") pod \"payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh\" (UID: \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\") " pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:34.644540 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.644517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9fa65c-d3ba-4290-8f9d-c22baac63554-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh\" (UID: \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\") " pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:34.652432 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.652404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxgh\" (UniqueName: \"kubernetes.io/projected/cc9fa65c-d3ba-4290-8f9d-c22baac63554-kube-api-access-pwxgh\") pod \"payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh\" (UID: \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\") " pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:34.844528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:34.844433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:35.117733 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:35.117638    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"]
Sep 08 13:48:35.122793 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:48:35.122760    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9fa65c_d3ba_4290_8f9d_c22baac63554.slice/crio-7b189846fa5a9a16734f8fe549ac3c9da2ff84500c9804471a7738c8b090957d WatchSource:0}: Error finding container 7b189846fa5a9a16734f8fe549ac3c9da2ff84500c9804471a7738c8b090957d: Status 404 returned error can't find the container with id 7b189846fa5a9a16734f8fe549ac3c9da2ff84500c9804471a7738c8b090957d
Sep 08 13:48:35.337729 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:35.337680    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh" event={"ID":"cc9fa65c-d3ba-4290-8f9d-c22baac63554","Type":"ContainerStarted","Data":"f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d"}
Sep 08 13:48:35.337729 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:35.337738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh" event={"ID":"cc9fa65c-d3ba-4290-8f9d-c22baac63554","Type":"ContainerStarted","Data":"7b189846fa5a9a16734f8fe549ac3c9da2ff84500c9804471a7738c8b090957d"}
Sep 08 13:48:35.337989 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:35.337763    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:35.357005 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:35.356948    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh" podStartSLOduration=1.3569293199999999 podStartE2EDuration="1.35692932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:35.355926709 +0000 UTC m=+1223.706912726" watchObservedRunningTime="2026-09-08 13:48:35.35692932 +0000 UTC m=+1223.707915267"
Sep 08 13:48:44.985334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:44.985291    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"]
Sep 08 13:48:46.060594 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:46.059617    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"]
Sep 08 13:48:46.187232 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:46.187202    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"]
Sep 08 13:48:46.187733 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:46.187690    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8" podUID="e8822de4-f6ef-4a30-b718-8d820d49ffef" containerName="istio-proxy" containerID="cri-o://368dd458bde5dbd84dbdd51efe96a39a0da4dd1ec40a596b8c016b1f68ae86e2" gracePeriod=30
Sep 08 13:48:46.346672 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:46.346576    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:48:47.266346 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:47.266306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"]
Sep 08 13:48:51.429504 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.429470    2561 generic.go:356] "Generic (PLEG): container finished" podID="e8822de4-f6ef-4a30-b718-8d820d49ffef" containerID="368dd458bde5dbd84dbdd51efe96a39a0da4dd1ec40a596b8c016b1f68ae86e2" exitCode=0
Sep 08 13:48:51.429878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.429516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8" event={"ID":"e8822de4-f6ef-4a30-b718-8d820d49ffef","Type":"ContainerDied","Data":"368dd458bde5dbd84dbdd51efe96a39a0da4dd1ec40a596b8c016b1f68ae86e2"}
Sep 08 13:48:51.429878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.429566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8" event={"ID":"e8822de4-f6ef-4a30-b718-8d820d49ffef","Type":"ContainerDied","Data":"e19148e7032320ddd59aa613842fc4310f2bf1302534f37e99bbe97f6e3c1dc8"}
Sep 08 13:48:51.429878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.429591    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e19148e7032320ddd59aa613842fc4310f2bf1302534f37e99bbe97f6e3c1dc8"
Sep 08 13:48:51.448522 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.448452    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:51.519689 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.519629    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhsfv\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-kube-api-access-dhsfv\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.519897 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.519724    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-data\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.519897 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.519780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-envoy\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.519897 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.519802    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-certs\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.519897 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.519873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-socket\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.520123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.519923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8822de4-f6ef-4a30-b718-8d820d49ffef-istiod-ca-cert\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.520123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.519952    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-token\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.520123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.519999    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-podinfo\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.520123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.520025    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-credential-socket\") pod \"e8822de4-f6ef-4a30-b718-8d820d49ffef\" (UID: \"e8822de4-f6ef-4a30-b718-8d820d49ffef\") "
Sep 08 13:48:51.520123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.520086    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:51.520323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.520130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:51.520412 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.520340    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.520512 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.520421    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.520512 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.520438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:51.520760 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.520739    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8822de4-f6ef-4a30-b718-8d820d49ffef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:48:51.520991 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.520969    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:51.522114 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.522083    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-kube-api-access-dhsfv" (OuterVolumeSpecName: "kube-api-access-dhsfv") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "kube-api-access-dhsfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:51.522398 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.522357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:51.522531 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.522368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:51.522796 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.522763    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8822de4-f6ef-4a30-b718-8d820d49ffef" (UID: "e8822de4-f6ef-4a30-b718-8d820d49ffef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:48:51.621608 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.621579    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8822de4-f6ef-4a30-b718-8d820d49ffef-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.621608 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.621609    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.623574 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.621618    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.623574 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.621626    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.623574 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.621636    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsfv\" (UniqueName: \"kubernetes.io/projected/e8822de4-f6ef-4a30-b718-8d820d49ffef-kube-api-access-dhsfv\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.623574 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.621645    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.623574 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:51.621653    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8822de4-f6ef-4a30-b718-8d820d49ffef-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:48:52.435514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.435475    2561 generic.go:356] "Generic (PLEG): container finished" podID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerID="e6b1016fa2cc756f7415938eedf212f4ac60d951991d507781fada38b5a292d9" exitCode=0
Sep 08 13:48:52.435947 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.435554    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" event={"ID":"afef1e25-eb30-4967-973f-bcbd04a7ab61","Type":"ContainerDied","Data":"e6b1016fa2cc756f7415938eedf212f4ac60d951991d507781fada38b5a292d9"}
Sep 08 13:48:52.435947 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.435629    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"
Sep 08 13:48:52.436112 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.436096    2561 scope.go:117] "RemoveContainer" containerID="e6b1016fa2cc756f7415938eedf212f4ac60d951991d507781fada38b5a292d9"
Sep 08 13:48:52.437182 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.437152    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:52.437305 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.437238    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:52.437305 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.437289    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:48:52.454690 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.454606    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"]
Sep 08 13:48:52.460015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:52.459977    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-dca93a-openshift-default-6596d589d6-2h7z8"]
Sep 08 13:48:53.447869 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:53.447811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" event={"ID":"afef1e25-eb30-4967-973f-bcbd04a7ab61","Type":"ContainerStarted","Data":"bd52cdaad836baac0f66f60f669d25568936004c1163b3410ca170d92fd093e3"}
Sep 08 13:48:53.817351 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:53.817252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:53.817594 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:53.817572    2561 patch_prober.go:28] interesting pod/e2e-mt-f8154283-openshift-default-687c55685b-48cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 08 13:48:53.817642 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:53.817619    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 08 13:48:54.269982 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:54.269934    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8822de4-f6ef-4a30-b718-8d820d49ffef" path="/var/lib/kubelet/pods/e8822de4-f6ef-4a30-b718-8d820d49ffef/volumes"
Sep 08 13:48:54.817296 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:54.817260    2561 patch_prober.go:28] interesting pod/e2e-mt-f8154283-openshift-default-687c55685b-48cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 08 13:48:54.817730 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:54.817318    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 08 13:48:55.817630 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:55.817580    2561 patch_prober.go:28] interesting pod/e2e-mt-f8154283-openshift-default-687c55685b-48cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 08 13:48:55.818157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:55.817651    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 08 13:48:56.817924 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:56.817880    2561 patch_prober.go:28] interesting pod/e2e-mt-f8154283-openshift-default-687c55685b-48cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 08 13:48:56.818468 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:56.817945    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 08 13:48:57.817793 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:57.817758    2561 patch_prober.go:28] interesting pod/e2e-mt-f8154283-openshift-default-687c55685b-48cbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 08 13:48:57.817951 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:57.817822    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 08 13:48:58.125094 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:58.125003    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"]
Sep 08 13:48:58.127760 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:58.127731    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2d1742-29814585-pms26"]
Sep 08 13:48:58.260694 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:58.260651    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b" path="/var/lib/kubelet/pods/907a5911-8ad8-4a6f-a2a3-9e4d7ad27a7b/volumes"
Sep 08 13:48:58.821554 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:58.821514    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:58.821951 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:58.821786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:48:58.822372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:48:58.822353    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:49:00.009458 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.009412    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn" podUID="9865bcc0-80d8-4e67-9467-b77987d8b0fd" containerName="main" containerID="cri-o://ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb" gracePeriod=45
Sep 08 13:49:00.254837 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.254811    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:49:00.419177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419143    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-home\") pod \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") "
Sep 08 13:49:00.419367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419218    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tmp-dir\") pod \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") "
Sep 08 13:49:00.419367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419250    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-dshm\") pod \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") "
Sep 08 13:49:00.419367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419280    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tls-certs\") pod \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") "
Sep 08 13:49:00.419367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419305    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-model-cache\") pod \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") "
Sep 08 13:49:00.419634 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419441    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g26hk\" (UniqueName: \"kubernetes.io/projected/9865bcc0-80d8-4e67-9467-b77987d8b0fd-kube-api-access-g26hk\") pod \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\" (UID: \"9865bcc0-80d8-4e67-9467-b77987d8b0fd\") "
Sep 08 13:49:00.419634 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419518    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-home" (OuterVolumeSpecName: "home") pod "9865bcc0-80d8-4e67-9467-b77987d8b0fd" (UID: "9865bcc0-80d8-4e67-9467-b77987d8b0fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:00.419634 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419581    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9865bcc0-80d8-4e67-9467-b77987d8b0fd" (UID: "9865bcc0-80d8-4e67-9467-b77987d8b0fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:00.419794 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419665    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "9865bcc0-80d8-4e67-9467-b77987d8b0fd" (UID: "9865bcc0-80d8-4e67-9467-b77987d8b0fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:00.419891 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419873    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:00.419943 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419897    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:00.419943 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.419911    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:00.421688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.421661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-dshm" (OuterVolumeSpecName: "dshm") pod "9865bcc0-80d8-4e67-9467-b77987d8b0fd" (UID: "9865bcc0-80d8-4e67-9467-b77987d8b0fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:00.421904 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.421883    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9865bcc0-80d8-4e67-9467-b77987d8b0fd" (UID: "9865bcc0-80d8-4e67-9467-b77987d8b0fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:49:00.421993 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.421971    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9865bcc0-80d8-4e67-9467-b77987d8b0fd-kube-api-access-g26hk" (OuterVolumeSpecName: "kube-api-access-g26hk") pod "9865bcc0-80d8-4e67-9467-b77987d8b0fd" (UID: "9865bcc0-80d8-4e67-9467-b77987d8b0fd"). InnerVolumeSpecName "kube-api-access-g26hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:00.487687 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.487652    2561 generic.go:356] "Generic (PLEG): container finished" podID="9865bcc0-80d8-4e67-9467-b77987d8b0fd" containerID="ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb" exitCode=0
Sep 08 13:49:00.487898 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.487699    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"
Sep 08 13:49:00.487898 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.487740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn" event={"ID":"9865bcc0-80d8-4e67-9467-b77987d8b0fd","Type":"ContainerDied","Data":"ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb"}
Sep 08 13:49:00.487898 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.487784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn" event={"ID":"9865bcc0-80d8-4e67-9467-b77987d8b0fd","Type":"ContainerDied","Data":"939959d157a879ccc71a29458373c8ee734bae26c5ed35928fa09bdc957ce236"}
Sep 08 13:49:00.487898 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.487804    2561 scope.go:117] "RemoveContainer" containerID="ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb"
Sep 08 13:49:00.497640 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.497620    2561 scope.go:117] "RemoveContainer" containerID="ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb"
Sep 08 13:49:00.497928 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:49:00.497910    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb\": container with ID starting with ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb not found: ID does not exist" containerID="ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb"
Sep 08 13:49:00.497988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.497937    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb"} err="failed to get container status \"ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb\": rpc error: code = NotFound desc = could not find container \"ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb\": container with ID starting with ce43dfad43660e372e756f29e8f2ff057db9c47b617c7497ce0ff24447eb5ffb not found: ID does not exist"
Sep 08 13:49:00.510266 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.510240    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"]
Sep 08 13:49:00.514359 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.514334    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2d1742/test-model-2d1742-kserve-86995fdf57-8z8xn"]
Sep 08 13:49:00.521390 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.521362    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26hk\" (UniqueName: \"kubernetes.io/projected/9865bcc0-80d8-4e67-9467-b77987d8b0fd-kube-api-access-g26hk\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:00.521528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.521418    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9865bcc0-80d8-4e67-9467-b77987d8b0fd-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:00.521528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:00.521436    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9865bcc0-80d8-4e67-9467-b77987d8b0fd-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:01.088506 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.088463    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f" podUID="8c103696-11ed-4af1-aa56-d98845630485" containerName="main" containerID="cri-o://26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310" gracePeriod=45
Sep 08 13:49:01.341854 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.341785    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:49:01.427811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.427773    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5brv5\" (UniqueName: \"kubernetes.io/projected/8c103696-11ed-4af1-aa56-d98845630485-kube-api-access-5brv5\") pod \"8c103696-11ed-4af1-aa56-d98845630485\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") "
Sep 08 13:49:01.427999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.427819    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-dshm\") pod \"8c103696-11ed-4af1-aa56-d98845630485\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") "
Sep 08 13:49:01.427999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.427839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c103696-11ed-4af1-aa56-d98845630485-tls-certs\") pod \"8c103696-11ed-4af1-aa56-d98845630485\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") "
Sep 08 13:49:01.427999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.427864    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-tmp-dir\") pod \"8c103696-11ed-4af1-aa56-d98845630485\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") "
Sep 08 13:49:01.427999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.427886    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-model-cache\") pod \"8c103696-11ed-4af1-aa56-d98845630485\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") "
Sep 08 13:49:01.427999 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.427922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-home\") pod \"8c103696-11ed-4af1-aa56-d98845630485\" (UID: \"8c103696-11ed-4af1-aa56-d98845630485\") "
Sep 08 13:49:01.428279 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.428151    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c103696-11ed-4af1-aa56-d98845630485" (UID: "8c103696-11ed-4af1-aa56-d98845630485"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:01.428279 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.428165    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c103696-11ed-4af1-aa56-d98845630485" (UID: "8c103696-11ed-4af1-aa56-d98845630485"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:01.428279 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.428257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-home" (OuterVolumeSpecName: "home") pod "8c103696-11ed-4af1-aa56-d98845630485" (UID: "8c103696-11ed-4af1-aa56-d98845630485"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:01.430097 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.430074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-dshm" (OuterVolumeSpecName: "dshm") pod "8c103696-11ed-4af1-aa56-d98845630485" (UID: "8c103696-11ed-4af1-aa56-d98845630485"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:01.430183 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.430116    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c103696-11ed-4af1-aa56-d98845630485-kube-api-access-5brv5" (OuterVolumeSpecName: "kube-api-access-5brv5") pod "8c103696-11ed-4af1-aa56-d98845630485" (UID: "8c103696-11ed-4af1-aa56-d98845630485"). InnerVolumeSpecName "kube-api-access-5brv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:01.430242 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.430186    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c103696-11ed-4af1-aa56-d98845630485-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c103696-11ed-4af1-aa56-d98845630485" (UID: "8c103696-11ed-4af1-aa56-d98845630485"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:49:01.493445 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.493400    2561 generic.go:356] "Generic (PLEG): container finished" podID="8c103696-11ed-4af1-aa56-d98845630485" containerID="26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310" exitCode=0
Sep 08 13:49:01.493445 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.493445    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"
Sep 08 13:49:01.493677 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.493481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f" event={"ID":"8c103696-11ed-4af1-aa56-d98845630485","Type":"ContainerDied","Data":"26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310"}
Sep 08 13:49:01.493677 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.493523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f" event={"ID":"8c103696-11ed-4af1-aa56-d98845630485","Type":"ContainerDied","Data":"aec5d1edfd50e9472ac734e849835fc9d269402f79b17d8841578bd8de70f816"}
Sep 08 13:49:01.493677 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.493545    2561 scope.go:117] "RemoveContainer" containerID="26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310"
Sep 08 13:49:01.506422 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.506335    2561 scope.go:117] "RemoveContainer" containerID="26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310"
Sep 08 13:49:01.507040 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:49:01.507017    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310\": container with ID starting with 26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310 not found: ID does not exist" containerID="26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310"
Sep 08 13:49:01.507149 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.507047    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310"} err="failed to get container status \"26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310\": rpc error: code = NotFound desc = could not find container \"26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310\": container with ID starting with 26f9632cedd9f10a7c8a36e5b29ffd3471172627cb84fac65fb584dad9f0a310 not found: ID does not exist"
Sep 08 13:49:01.517001 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.516970    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"]
Sep 08 13:49:01.519153 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.519125    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-dca93a/noten-model-dca93a-kserve-6c8bcb55bc-9rw8f"]
Sep 08 13:49:01.528946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.528921    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:01.528946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.528944    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5brv5\" (UniqueName: \"kubernetes.io/projected/8c103696-11ed-4af1-aa56-d98845630485-kube-api-access-5brv5\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:01.529110 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.528954    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:01.529110 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.528964    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c103696-11ed-4af1-aa56-d98845630485-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:01.529110 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.528973    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:01.529110 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:01.528981    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c103696-11ed-4af1-aa56-d98845630485-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:02.244716 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.244682    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c103696-11ed-4af1-aa56-d98845630485" path="/var/lib/kubelet/pods/8c103696-11ed-4af1-aa56-d98845630485/volumes"
Sep 08 13:49:02.245110 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.245075    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9865bcc0-80d8-4e67-9467-b77987d8b0fd" path="/var/lib/kubelet/pods/9865bcc0-80d8-4e67-9467-b77987d8b0fd/volumes"
Sep 08 13:49:02.286575 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.286537    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj" podUID="1e6fff33-996e-4f6b-ace3-c3e82fb18beb" containerName="main" containerID="cri-o://d0194dde51be24a3862af1eff2434ffbee2317708ff445248c53fd6bf4143115" gracePeriod=45
Sep 08 13:49:02.501495 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.501406    2561 generic.go:356] "Generic (PLEG): container finished" podID="1e6fff33-996e-4f6b-ace3-c3e82fb18beb" containerID="d0194dde51be24a3862af1eff2434ffbee2317708ff445248c53fd6bf4143115" exitCode=0
Sep 08 13:49:02.501495 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.501479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj" event={"ID":"1e6fff33-996e-4f6b-ace3-c3e82fb18beb","Type":"ContainerDied","Data":"d0194dde51be24a3862af1eff2434ffbee2317708ff445248c53fd6bf4143115"}
Sep 08 13:49:02.540169 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.540145    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:49:02.636210 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636173    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm9rm\" (UniqueName: \"kubernetes.io/projected/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-kube-api-access-cm9rm\") pod \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") "
Sep 08 13:49:02.636427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tmp-dir\") pod \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") "
Sep 08 13:49:02.636427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636256    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-model-cache\") pod \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") "
Sep 08 13:49:02.636427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636276    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-dshm\") pod \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") "
Sep 08 13:49:02.636427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tls-certs\") pod \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") "
Sep 08 13:49:02.636427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636410    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-home\") pod \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\" (UID: \"1e6fff33-996e-4f6b-ace3-c3e82fb18beb\") "
Sep 08 13:49:02.636724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636587    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e6fff33-996e-4f6b-ace3-c3e82fb18beb" (UID: "1e6fff33-996e-4f6b-ace3-c3e82fb18beb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:02.636724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636612    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e6fff33-996e-4f6b-ace3-c3e82fb18beb" (UID: "1e6fff33-996e-4f6b-ace3-c3e82fb18beb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:02.636724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636679    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:02.636724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.636688    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-home" (OuterVolumeSpecName: "home") pod "1e6fff33-996e-4f6b-ace3-c3e82fb18beb" (UID: "1e6fff33-996e-4f6b-ace3-c3e82fb18beb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:02.638635 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.638603    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e6fff33-996e-4f6b-ace3-c3e82fb18beb" (UID: "1e6fff33-996e-4f6b-ace3-c3e82fb18beb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:49:02.638746 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.638611    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-dshm" (OuterVolumeSpecName: "dshm") pod "1e6fff33-996e-4f6b-ace3-c3e82fb18beb" (UID: "1e6fff33-996e-4f6b-ace3-c3e82fb18beb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:02.638746 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.638678    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-kube-api-access-cm9rm" (OuterVolumeSpecName: "kube-api-access-cm9rm") pod "1e6fff33-996e-4f6b-ace3-c3e82fb18beb" (UID: "1e6fff33-996e-4f6b-ace3-c3e82fb18beb"). InnerVolumeSpecName "kube-api-access-cm9rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:02.737183 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.737142    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:02.737367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.737199    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:02.737367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.737210    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:02.737367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.737218    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:02.737367 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:02.737227    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm9rm\" (UniqueName: \"kubernetes.io/projected/1e6fff33-996e-4f6b-ace3-c3e82fb18beb-kube-api-access-cm9rm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:03.508482 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:03.508451    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"
Sep 08 13:49:03.508946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:03.508453    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj" event={"ID":"1e6fff33-996e-4f6b-ace3-c3e82fb18beb","Type":"ContainerDied","Data":"0330e206709d79001bf80cbc74173bc0a972ffd4076e28aec4a65a8fe0c48576"}
Sep 08 13:49:03.508946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:03.508576    2561 scope.go:117] "RemoveContainer" containerID="d0194dde51be24a3862af1eff2434ffbee2317708ff445248c53fd6bf4143115"
Sep 08 13:49:03.532046 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:03.532012    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"]
Sep 08 13:49:03.538478 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:03.538454    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a7ef4b/test-model-a7ef4b-kserve-6cdb576d99-8m7jj"]
Sep 08 13:49:04.244797 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:04.244765    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e6fff33-996e-4f6b-ace3-c3e82fb18beb" path="/var/lib/kubelet/pods/1e6fff33-996e-4f6b-ace3-c3e82fb18beb/volumes"
Sep 08 13:49:05.649177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649139    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"]
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649451    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9865bcc0-80d8-4e67-9467-b77987d8b0fd" containerName="main"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649462    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9865bcc0-80d8-4e67-9467-b77987d8b0fd" containerName="main"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649470    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c103696-11ed-4af1-aa56-d98845630485" containerName="main"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649476    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c103696-11ed-4af1-aa56-d98845630485" containerName="main"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649490    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8822de4-f6ef-4a30-b718-8d820d49ffef" containerName="istio-proxy"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649495    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8822de4-f6ef-4a30-b718-8d820d49ffef" containerName="istio-proxy"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649503    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6fff33-996e-4f6b-ace3-c3e82fb18beb" containerName="main"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649509    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6fff33-996e-4f6b-ace3-c3e82fb18beb" containerName="main"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649573    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9865bcc0-80d8-4e67-9467-b77987d8b0fd" containerName="main"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649582    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8822de4-f6ef-4a30-b718-8d820d49ffef" containerName="istio-proxy"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649590    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c103696-11ed-4af1-aa56-d98845630485" containerName="main"
Sep 08 13:49:05.649656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.649600    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6fff33-996e-4f6b-ace3-c3e82fb18beb" containerName="main"
Sep 08 13:49:05.653723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.653702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"
Sep 08 13:49:05.656344 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.656321    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 13:49:05.656586 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.656570    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 13:49:05.660209 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.660174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b316d95d-dec8-4892-a0d4-e403ab9fdf2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q\" (UID: \"b316d95d-dec8-4892-a0d4-e403ab9fdf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"
Sep 08 13:49:05.660518 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.660491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"]
Sep 08 13:49:05.760935 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.760888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b316d95d-dec8-4892-a0d4-e403ab9fdf2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q\" (UID: \"b316d95d-dec8-4892-a0d4-e403ab9fdf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"
Sep 08 13:49:05.761539 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.761521    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b316d95d-dec8-4892-a0d4-e403ab9fdf2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q\" (UID: \"b316d95d-dec8-4892-a0d4-e403ab9fdf2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"
Sep 08 13:49:05.965230 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:05.965194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"
Sep 08 13:49:06.235484 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:06.235457    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"]
Sep 08 13:49:06.237893 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:49:06.237864    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb316d95d_dec8_4892_a0d4_e403ab9fdf2d.slice/crio-95ae457a592af35a140c8b77c1e9b5412b9119831e627efb230dbf7c82fe3d73 WatchSource:0}: Error finding container 95ae457a592af35a140c8b77c1e9b5412b9119831e627efb230dbf7c82fe3d73: Status 404 returned error can't find the container with id 95ae457a592af35a140c8b77c1e9b5412b9119831e627efb230dbf7c82fe3d73
Sep 08 13:49:06.525479 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:06.525364    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q" event={"ID":"b316d95d-dec8-4892-a0d4-e403ab9fdf2d","Type":"ContainerStarted","Data":"95ae457a592af35a140c8b77c1e9b5412b9119831e627efb230dbf7c82fe3d73"}
Sep 08 13:49:07.532094 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:07.532057    2561 generic.go:356] "Generic (PLEG): container finished" podID="b316d95d-dec8-4892-a0d4-e403ab9fdf2d" containerID="37ce922be4bfaf12eaad533b032207d948da3d47e8d92af8a81d683ba348820a" exitCode=0
Sep 08 13:49:07.532566 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:07.532139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q" event={"ID":"b316d95d-dec8-4892-a0d4-e403ab9fdf2d","Type":"ContainerDied","Data":"37ce922be4bfaf12eaad533b032207d948da3d47e8d92af8a81d683ba348820a"}
Sep 08 13:49:08.672501 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:08.672477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"
Sep 08 13:49:08.687431 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:08.687404    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b316d95d-dec8-4892-a0d4-e403ab9fdf2d-maas-api-service-ca\") pod \"b316d95d-dec8-4892-a0d4-e403ab9fdf2d\" (UID: \"b316d95d-dec8-4892-a0d4-e403ab9fdf2d\") "
Sep 08 13:49:08.687740 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:08.687718    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b316d95d-dec8-4892-a0d4-e403ab9fdf2d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b316d95d-dec8-4892-a0d4-e403ab9fdf2d" (UID: "b316d95d-dec8-4892-a0d4-e403ab9fdf2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:08.788611 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:08.788524    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b316d95d-dec8-4892-a0d4-e403ab9fdf2d-maas-api-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:09.542802 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:09.542770    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"
Sep 08 13:49:09.542977 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:09.542804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q" event={"ID":"b316d95d-dec8-4892-a0d4-e403ab9fdf2d","Type":"ContainerDied","Data":"95ae457a592af35a140c8b77c1e9b5412b9119831e627efb230dbf7c82fe3d73"}
Sep 08 13:49:09.542977 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:09.542853    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95ae457a592af35a140c8b77c1e9b5412b9119831e627efb230dbf7c82fe3d73"
Sep 08 13:49:16.150974 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.150881    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"]
Sep 08 13:49:16.151459 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.151167    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh" podUID="cc9fa65c-d3ba-4290-8f9d-c22baac63554" containerName="payload-processing" containerID="cri-o://f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d" gracePeriod=30
Sep 08 13:49:16.406799 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.406731    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:49:16.456738 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.456704    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9fa65c-d3ba-4290-8f9d-c22baac63554-plugins-config-volume\") pod \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\" (UID: \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\") "
Sep 08 13:49:16.456912 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.456760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwxgh\" (UniqueName: \"kubernetes.io/projected/cc9fa65c-d3ba-4290-8f9d-c22baac63554-kube-api-access-pwxgh\") pod \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\" (UID: \"cc9fa65c-d3ba-4290-8f9d-c22baac63554\") "
Sep 08 13:49:16.457128 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.457106    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9fa65c-d3ba-4290-8f9d-c22baac63554-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc9fa65c-d3ba-4290-8f9d-c22baac63554" (UID: "cc9fa65c-d3ba-4290-8f9d-c22baac63554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:16.458848 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.458823    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9fa65c-d3ba-4290-8f9d-c22baac63554-kube-api-access-pwxgh" (OuterVolumeSpecName: "kube-api-access-pwxgh") pod "cc9fa65c-d3ba-4290-8f9d-c22baac63554" (UID: "cc9fa65c-d3ba-4290-8f9d-c22baac63554"). InnerVolumeSpecName "kube-api-access-pwxgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:16.517722 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.517685    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"]
Sep 08 13:49:16.520640 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.520612    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a7ef4b-8f72q"]
Sep 08 13:49:16.557997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.557958    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwxgh\" (UniqueName: \"kubernetes.io/projected/cc9fa65c-d3ba-4290-8f9d-c22baac63554-kube-api-access-pwxgh\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:16.557997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.557997    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9fa65c-d3ba-4290-8f9d-c22baac63554-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:16.576266 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.576233    2561 generic.go:356] "Generic (PLEG): container finished" podID="cc9fa65c-d3ba-4290-8f9d-c22baac63554" containerID="f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d" exitCode=0
Sep 08 13:49:16.576449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.576278    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"
Sep 08 13:49:16.576449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.576323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh" event={"ID":"cc9fa65c-d3ba-4290-8f9d-c22baac63554","Type":"ContainerDied","Data":"f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d"}
Sep 08 13:49:16.576449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.576392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh" event={"ID":"cc9fa65c-d3ba-4290-8f9d-c22baac63554","Type":"ContainerDied","Data":"7b189846fa5a9a16734f8fe549ac3c9da2ff84500c9804471a7738c8b090957d"}
Sep 08 13:49:16.576449 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.576424    2561 scope.go:117] "RemoveContainer" containerID="f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d"
Sep 08 13:49:16.585544 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.585525    2561 scope.go:117] "RemoveContainer" containerID="f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d"
Sep 08 13:49:16.585811 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:49:16.585795    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d\": container with ID starting with f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d not found: ID does not exist" containerID="f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d"
Sep 08 13:49:16.585902 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.585819    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d"} err="failed to get container status \"f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d\": rpc error: code = NotFound desc = could not find container \"f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d\": container with ID starting with f740f7984731f4125b5739de8c404e950c55eed6c248523fde87dd2a35c5ba3d not found: ID does not exist"
Sep 08 13:49:16.598678 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.598652    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"]
Sep 08 13:49:16.605593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:16.605567    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh"]
Sep 08 13:49:17.343686 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:17.343639    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f8154283-85f5d5b4f-sssnh" podUID="cc9fa65c-d3ba-4290-8f9d-c22baac63554" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.62:9004: i/o timeout"
Sep 08 13:49:18.244472 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:18.244440    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b316d95d-dec8-4892-a0d4-e403ab9fdf2d" path="/var/lib/kubelet/pods/b316d95d-dec8-4892-a0d4-e403ab9fdf2d/volumes"
Sep 08 13:49:18.244822 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:18.244808    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9fa65c-d3ba-4290-8f9d-c22baac63554" path="/var/lib/kubelet/pods/cc9fa65c-d3ba-4290-8f9d-c22baac63554/volumes"
Sep 08 13:49:23.473713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:23.473669    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"]
Sep 08 13:49:23.474203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:23.473971    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy" containerID="cri-o://bd52cdaad836baac0f66f60f669d25568936004c1163b3410ca170d92fd093e3" gracePeriod=30
Sep 08 13:49:24.041268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:24.041236    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"]
Sep 08 13:49:24.041483 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:24.041459    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy" containerID="cri-o://bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b" gracePeriod=30
Sep 08 13:49:25.036649 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.036614    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"]
Sep 08 13:49:25.037177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.036907    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b316d95d-dec8-4892-a0d4-e403ab9fdf2d" containerName="revoke-keys"
Sep 08 13:49:25.037177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.036919    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b316d95d-dec8-4892-a0d4-e403ab9fdf2d" containerName="revoke-keys"
Sep 08 13:49:25.037177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.036954    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9fa65c-d3ba-4290-8f9d-c22baac63554" containerName="payload-processing"
Sep 08 13:49:25.037177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.036960    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9fa65c-d3ba-4290-8f9d-c22baac63554" containerName="payload-processing"
Sep 08 13:49:25.037177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.037019    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9fa65c-d3ba-4290-8f9d-c22baac63554" containerName="payload-processing"
Sep 08 13:49:25.037177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.037026    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b316d95d-dec8-4892-a0d4-e403ab9fdf2d" containerName="revoke-keys"
Sep 08 13:49:25.041293 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.041271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.044129 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.044111    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1f082e-openshift-default-dockercfg-jvs9r\""
Sep 08 13:49:25.055290 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.055263    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"]
Sep 08 13:49:25.133428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-token\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.133625 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-data\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.133625 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl9j8\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-kube-api-access-gl9j8\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.133625 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-socket\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.133625 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-credential-socket\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.133777 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-certs\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.133777 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50bf45d-2f90-4d53-bb87-6b8d98246502-istiod-ca-cert\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.133777 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-envoy\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.133888 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.133779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-podinfo\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.234737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.234691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-token\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.234946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.234747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-data\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.234946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.234920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl9j8\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-kube-api-access-gl9j8\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235063 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.234977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-socket\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235063 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-credential-socket\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235063 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-certs\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235291 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50bf45d-2f90-4d53-bb87-6b8d98246502-istiod-ca-cert\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235291 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-envoy\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235291 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-data\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235291 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-podinfo\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235536 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-certs\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235536 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-socket\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235536 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235469    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-credential-socket\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.235787 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.235764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50bf45d-2f90-4d53-bb87-6b8d98246502-istiod-ca-cert\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.237430 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.237411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-envoy\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.237653 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.237636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-podinfo\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.243575 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.243545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-token\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.243735 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.243717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl9j8\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-kube-api-access-gl9j8\") pod \"e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") " pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.351569 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.351484    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:25.626760 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.626735    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"]
Sep 08 13:49:25.629117 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:49:25.629087    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50bf45d_2f90_4d53_bb87_6b8d98246502.slice/crio-bd47b30a3edeab2a24c0b39880df4494bb7f244aebef4d47d122699932ff6bf1 WatchSource:0}: Error finding container bd47b30a3edeab2a24c0b39880df4494bb7f244aebef4d47d122699932ff6bf1: Status 404 returned error can't find the container with id bd47b30a3edeab2a24c0b39880df4494bb7f244aebef4d47d122699932ff6bf1
Sep 08 13:49:25.631328 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.631294    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:25.631404 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.631363    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:25.631466 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:25.631414    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:26.626334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:26.626291    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" event={"ID":"f50bf45d-2f90-4d53-bb87-6b8d98246502","Type":"ContainerStarted","Data":"2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9"}
Sep 08 13:49:26.626746 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:26.626350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" event={"ID":"f50bf45d-2f90-4d53-bb87-6b8d98246502","Type":"ContainerStarted","Data":"bd47b30a3edeab2a24c0b39880df4494bb7f244aebef4d47d122699932ff6bf1"}
Sep 08 13:49:26.650069 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:26.650022    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podStartSLOduration=1.6500082759999999 podStartE2EDuration="1.650008276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:49:26.64869251 +0000 UTC m=+1274.999678459" watchObservedRunningTime="2026-09-08 13:49:26.650008276 +0000 UTC m=+1275.000994223"
Sep 08 13:49:27.352645 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:27.352606    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:27.357600 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:27.357575    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:27.631526 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:27.631442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:27.632633 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:27.632612    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:28.639262 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.639229    2561 generic.go:356] "Generic (PLEG): container finished" podID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerID="bd52cdaad836baac0f66f60f669d25568936004c1163b3410ca170d92fd093e3" exitCode=0
Sep 08 13:49:28.639638 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.639304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" event={"ID":"afef1e25-eb30-4967-973f-bcbd04a7ab61","Type":"ContainerDied","Data":"bd52cdaad836baac0f66f60f669d25568936004c1163b3410ca170d92fd093e3"}
Sep 08 13:49:28.639638 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.639390    2561 scope.go:117] "RemoveContainer" containerID="e6b1016fa2cc756f7415938eedf212f4ac60d951991d507781fada38b5a292d9"
Sep 08 13:49:28.725844 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.725816    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:49:28.865899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.865799    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-token\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.865899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.865839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-socket\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.865899 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.865894    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-podinfo\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.866152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.865974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgsj9\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-kube-api-access-qgsj9\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.866152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866006    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-data\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.866152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-certs\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.866152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866058    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afef1e25-eb30-4967-973f-bcbd04a7ab61-istiod-ca-cert\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.866152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-envoy\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.866152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866112    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-credential-socket\") pod \"afef1e25-eb30-4967-973f-bcbd04a7ab61\" (UID: \"afef1e25-eb30-4967-973f-bcbd04a7ab61\") "
Sep 08 13:49:28.866476 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866156    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:28.866476 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866463    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:28.866591 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866512    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:28.866591 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afef1e25-eb30-4967-973f-bcbd04a7ab61-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:28.866707 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866669    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:28.866770 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.866733    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-data" (OuterVolumeSpecName: "istio-data") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:28.868167 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.868138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:49:28.868167 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.868149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-kube-api-access-qgsj9" (OuterVolumeSpecName: "kube-api-access-qgsj9") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "kube-api-access-qgsj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:28.868455 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.868439    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-token" (OuterVolumeSpecName: "istio-token") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:28.868513 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.868484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afef1e25-eb30-4967-973f-bcbd04a7ab61" (UID: "afef1e25-eb30-4967-973f-bcbd04a7ab61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:28.967798 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.967759    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:28.967798 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.967790    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afef1e25-eb30-4967-973f-bcbd04a7ab61-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:28.967798 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.967801    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:28.968033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.967812    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:28.968033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.967822    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:28.968033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.967830    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:28.968033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.967838    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgsj9\" (UniqueName: \"kubernetes.io/projected/afef1e25-eb30-4967-973f-bcbd04a7ab61-kube-api-access-qgsj9\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:28.968033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:28.967846    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afef1e25-eb30-4967-973f-bcbd04a7ab61-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.289465 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.289437    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:49:29.371450 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hksbm\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-kube-api-access-hksbm\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.371681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-token\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.371681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371482    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-data\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.371681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-certs\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.371681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371564    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-socket\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.371681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371590    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-credential-socket\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.371681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371658    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-envoy\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.372015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371699    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-podinfo\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.372015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istiod-ca-cert\") pod \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\" (UID: \"df004aa6-08ae-47f7-a6b3-c7468e6699f4\") "
Sep 08 13:49:29.372015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371894    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:29.372015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.371875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:29.372015 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.372002    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:29.372253 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.372080    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:29.372310 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.372267    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:29.372372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.372347    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.372372 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.372366    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.372509 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.372398    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.372509 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.372412    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.372509 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.372426    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.373976 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.373950    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:29.374219 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.374202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-kube-api-access-hksbm" (OuterVolumeSpecName: "kube-api-access-hksbm") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "kube-api-access-hksbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:29.374475 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.374447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:29.374475 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.374454    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df004aa6-08ae-47f7-a6b3-c7468e6699f4" (UID: "df004aa6-08ae-47f7-a6b3-c7468e6699f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:49:29.473033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.472996    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.473033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.473028    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.473033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.473037    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hksbm\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-kube-api-access-hksbm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.473266 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.473049    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df004aa6-08ae-47f7-a6b3-c7468e6699f4-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:49:29.651563 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.651521    2561 generic.go:356] "Generic (PLEG): container finished" podID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerID="bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b" exitCode=0
Sep 08 13:49:29.652056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.651581    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"
Sep 08 13:49:29.652056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.651608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" event={"ID":"df004aa6-08ae-47f7-a6b3-c7468e6699f4","Type":"ContainerDied","Data":"bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b"}
Sep 08 13:49:29.652056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.651693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m" event={"ID":"df004aa6-08ae-47f7-a6b3-c7468e6699f4","Type":"ContainerDied","Data":"c974de5e1b6cb87496e403b6dda8fd8f604bce928872c92be43e84ba613b303e"}
Sep 08 13:49:29.652056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.651719    2561 scope.go:117] "RemoveContainer" containerID="bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b"
Sep 08 13:49:29.653454 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.653350    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"
Sep 08 13:49:29.653454 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.653369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg" event={"ID":"afef1e25-eb30-4967-973f-bcbd04a7ab61","Type":"ContainerDied","Data":"b07500cba2fbc0e636b38c00ee9580aa13dc36756a31461ec102de8f3910f6aa"}
Sep 08 13:49:29.662509 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.662488    2561 scope.go:117] "RemoveContainer" containerID="ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5"
Sep 08 13:49:29.671307 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.671285    2561 scope.go:117] "RemoveContainer" containerID="bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b"
Sep 08 13:49:29.671670 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:49:29.671647    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b\": container with ID starting with bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b not found: ID does not exist" containerID="bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b"
Sep 08 13:49:29.671753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.671680    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b"} err="failed to get container status \"bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b\": rpc error: code = NotFound desc = could not find container \"bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b\": container with ID starting with bf1768883f248f97b1bc11ef1b38cd3faa2df07d0390c089ac07eba3acbe789b not found: ID does not exist"
Sep 08 13:49:29.671753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.671700    2561 scope.go:117] "RemoveContainer" containerID="ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5"
Sep 08 13:49:29.671988 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:49:29.671968    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5\": container with ID starting with ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5 not found: ID does not exist" containerID="ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5"
Sep 08 13:49:29.672056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.672000    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5"} err="failed to get container status \"ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5\": rpc error: code = NotFound desc = could not find container \"ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5\": container with ID starting with ffffc990aff45d7b9980d1d4d922991a50ecec2ffd1e2f6c34693d849acbf3e5 not found: ID does not exist"
Sep 08 13:49:29.672056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.672025    2561 scope.go:117] "RemoveContainer" containerID="bd52cdaad836baac0f66f60f669d25568936004c1163b3410ca170d92fd093e3"
Sep 08 13:49:29.682848 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.682816    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"]
Sep 08 13:49:29.684351 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.684323    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a7ef4b-openshift-default-db86d7b58-jf99m"]
Sep 08 13:49:29.699995 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.699960    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"]
Sep 08 13:49:29.704203 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:29.704171    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f8154283-openshift-default-687c55685b-48cbg"]
Sep 08 13:49:30.245134 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.245087    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" path="/var/lib/kubelet/pods/afef1e25-eb30-4967-973f-bcbd04a7ab61/volumes"
Sep 08 13:49:30.245623 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.245604    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" path="/var/lib/kubelet/pods/df004aa6-08ae-47f7-a6b3-c7468e6699f4/volumes"
Sep 08 13:49:30.694296 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694256    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"]
Sep 08 13:49:30.694791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694675    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy"
Sep 08 13:49:30.694791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694694    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy"
Sep 08 13:49:30.694791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694711    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy"
Sep 08 13:49:30.694791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694718    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy"
Sep 08 13:49:30.694791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694734    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy"
Sep 08 13:49:30.694791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694742    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy"
Sep 08 13:49:30.694791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694760    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy"
Sep 08 13:49:30.694791 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694769    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy"
Sep 08 13:49:30.695186 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694839    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy"
Sep 08 13:49:30.695186 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694850    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy"
Sep 08 13:49:30.695186 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.694860    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="df004aa6-08ae-47f7-a6b3-c7468e6699f4" containerName="istio-proxy"
Sep 08 13:49:30.699268 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.699241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:30.701878 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.701854    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1f082e\""
Sep 08 13:49:30.707293 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.707259    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"]
Sep 08 13:49:30.720528 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.720493    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"]
Sep 08 13:49:30.720882 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.720867    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="afef1e25-eb30-4967-973f-bcbd04a7ab61" containerName="istio-proxy"
Sep 08 13:49:30.723756 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.723732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:30.726283 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.726259    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1f082e-dockercfg-j4gbb\""
Sep 08 13:49:30.738625 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.738593    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"]
Sep 08 13:49:30.784093 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.784050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvvd\" (UniqueName: \"kubernetes.io/projected/f9aba5bf-0101-4121-a8dc-4933bffd79cb-kube-api-access-psvvd\") pod \"payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88\" (UID: \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:30.784254 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.784103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v\" (UID: \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:30.784254 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.784164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ddz\" (UniqueName: \"kubernetes.io/projected/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-kube-api-access-q4ddz\") pod \"payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v\" (UID: \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:30.784254 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.784236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9aba5bf-0101-4121-a8dc-4933bffd79cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88\" (UID: \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:30.885356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.885314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvvd\" (UniqueName: \"kubernetes.io/projected/f9aba5bf-0101-4121-a8dc-4933bffd79cb-kube-api-access-psvvd\") pod \"payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88\" (UID: \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:30.885550 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.885369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v\" (UID: \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:30.885550 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.885421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ddz\" (UniqueName: \"kubernetes.io/projected/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-kube-api-access-q4ddz\") pod \"payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v\" (UID: \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:30.885550 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.885455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9aba5bf-0101-4121-a8dc-4933bffd79cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88\" (UID: \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:30.886197 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.886176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9aba5bf-0101-4121-a8dc-4933bffd79cb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88\" (UID: \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:30.886234 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.886178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v\" (UID: \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:30.893937 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.893901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ddz\" (UniqueName: \"kubernetes.io/projected/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-kube-api-access-q4ddz\") pod \"payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v\" (UID: \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:30.894099 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:30.893903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvvd\" (UniqueName: \"kubernetes.io/projected/f9aba5bf-0101-4121-a8dc-4933bffd79cb-kube-api-access-psvvd\") pod \"payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88\" (UID: \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:31.010103 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.010004    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:31.034334 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.034288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:31.288843 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.288730    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"]
Sep 08 13:49:31.291535 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:49:31.291511    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e15d50_e9f4_46a2_a8eb_c0e7378e8479.slice/crio-64ac72a15748e5b88f7b7811028b47de53216210e9f440f8f8dbc65e62275eab WatchSource:0}: Error finding container 64ac72a15748e5b88f7b7811028b47de53216210e9f440f8f8dbc65e62275eab: Status 404 returned error can't find the container with id 64ac72a15748e5b88f7b7811028b47de53216210e9f440f8f8dbc65e62275eab
Sep 08 13:49:31.326213 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.326182    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"]
Sep 08 13:49:31.330064 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:49:31.330033    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9aba5bf_0101_4121_a8dc_4933bffd79cb.slice/crio-44a3b085300589adc5aca22bc46d09295cd58ab87baabf52e6f2c3c64ff2bce5 WatchSource:0}: Error finding container 44a3b085300589adc5aca22bc46d09295cd58ab87baabf52e6f2c3c64ff2bce5: Status 404 returned error can't find the container with id 44a3b085300589adc5aca22bc46d09295cd58ab87baabf52e6f2c3c64ff2bce5
Sep 08 13:49:31.667493 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.667362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v" event={"ID":"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479","Type":"ContainerStarted","Data":"89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291"}
Sep 08 13:49:31.667493 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.667455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v" event={"ID":"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479","Type":"ContainerStarted","Data":"64ac72a15748e5b88f7b7811028b47de53216210e9f440f8f8dbc65e62275eab"}
Sep 08 13:49:31.667493 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.667490    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:31.668971 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.668940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88" event={"ID":"f9aba5bf-0101-4121-a8dc-4933bffd79cb","Type":"ContainerStarted","Data":"de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2"}
Sep 08 13:49:31.669113 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.668985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88" event={"ID":"f9aba5bf-0101-4121-a8dc-4933bffd79cb","Type":"ContainerStarted","Data":"44a3b085300589adc5aca22bc46d09295cd58ab87baabf52e6f2c3c64ff2bce5"}
Sep 08 13:49:31.669113 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.669027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:31.686607 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.686546    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v" podStartSLOduration=1.6865280089999999 podStartE2EDuration="1.686528009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:49:31.686104018 +0000 UTC m=+1280.037089981" watchObservedRunningTime="2026-09-08 13:49:31.686528009 +0000 UTC m=+1280.037513958"
Sep 08 13:49:31.706127 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:31.706063    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88" podStartSLOduration=1.706046378 podStartE2EDuration="1.706046378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:49:31.705582754 +0000 UTC m=+1280.056568702" watchObservedRunningTime="2026-09-08 13:49:31.706046378 +0000 UTC m=+1280.057032384"
Sep 08 13:49:42.677637 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:42.677514    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 13:49:42.679658 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:42.679621    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 13:49:43.673627 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:43.673587    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 13:49:43.675796 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:43.673652    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 13:49:44.194117 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.194072    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"]
Sep 08 13:49:44.198010 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.197976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:44.201151 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.201119    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-3c9fc8-dockercfg-c7jvf\""
Sep 08 13:49:44.201327 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.201162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-3c9fc8\""
Sep 08 13:49:44.205262 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.205220    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"]
Sep 08 13:49:44.304842 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.304803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7j2r\" (UniqueName: \"kubernetes.io/projected/4cabf638-c92a-4e30-b850-04babd96452c-kube-api-access-v7j2r\") pod \"payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr\" (UID: \"4cabf638-c92a-4e30-b850-04babd96452c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:44.305059 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.304871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cabf638-c92a-4e30-b850-04babd96452c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr\" (UID: \"4cabf638-c92a-4e30-b850-04babd96452c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:44.406330 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.406272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7j2r\" (UniqueName: \"kubernetes.io/projected/4cabf638-c92a-4e30-b850-04babd96452c-kube-api-access-v7j2r\") pod \"payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr\" (UID: \"4cabf638-c92a-4e30-b850-04babd96452c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:44.406587 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.406370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cabf638-c92a-4e30-b850-04babd96452c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr\" (UID: \"4cabf638-c92a-4e30-b850-04babd96452c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:44.407400 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.407346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cabf638-c92a-4e30-b850-04babd96452c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr\" (UID: \"4cabf638-c92a-4e30-b850-04babd96452c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:44.420105 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.420076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7j2r\" (UniqueName: \"kubernetes.io/projected/4cabf638-c92a-4e30-b850-04babd96452c-kube-api-access-v7j2r\") pod \"payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr\" (UID: \"4cabf638-c92a-4e30-b850-04babd96452c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:44.510551 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.510437    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:44.740033 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.739997    2561 generic.go:356] "Generic (PLEG): container finished" podID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerID="2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9" exitCode=0
Sep 08 13:49:44.740241 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.740062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" event={"ID":"f50bf45d-2f90-4d53-bb87-6b8d98246502","Type":"ContainerDied","Data":"2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9"}
Sep 08 13:49:44.740478 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.740460    2561 scope.go:117] "RemoveContainer" containerID="2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9"
Sep 08 13:49:44.741576 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.741543    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:44.741656 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.741632    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:44.741710 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.741679    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:44.812883 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:44.812829    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"]
Sep 08 13:49:44.817642 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:49:44.817601    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cabf638_c92a_4e30_b850_04babd96452c.slice/crio-b45268d3652297a5520fcf02ef76aec869752b090f00bba89e09f18b03757d0c WatchSource:0}: Error finding container b45268d3652297a5520fcf02ef76aec869752b090f00bba89e09f18b03757d0c: Status 404 returned error can't find the container with id b45268d3652297a5520fcf02ef76aec869752b090f00bba89e09f18b03757d0c
Sep 08 13:49:45.755075 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:45.755023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" event={"ID":"f50bf45d-2f90-4d53-bb87-6b8d98246502","Type":"ContainerStarted","Data":"b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec"}
Sep 08 13:49:45.757659 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:45.757616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr" event={"ID":"4cabf638-c92a-4e30-b850-04babd96452c","Type":"ContainerStarted","Data":"efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434"}
Sep 08 13:49:45.757801 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:45.757689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr" event={"ID":"4cabf638-c92a-4e30-b850-04babd96452c","Type":"ContainerStarted","Data":"b45268d3652297a5520fcf02ef76aec869752b090f00bba89e09f18b03757d0c"}
Sep 08 13:49:45.757801 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:45.757723    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:45.796710 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:45.796653    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr" podStartSLOduration=1.796637297 podStartE2EDuration="1.796637297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:49:45.795729 +0000 UTC m=+1294.146714954" watchObservedRunningTime="2026-09-08 13:49:45.796637297 +0000 UTC m=+1294.147623245"
Sep 08 13:49:46.352610 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:46.352572    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:46.352962 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:46.352937    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 13:49:46.353164 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:46.353138    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 13:49:47.352279 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:47.352225    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 13:49:47.352869 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:47.352306    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 13:49:48.352531 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:48.352486    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 13:49:48.353143 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:48.353105    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 13:49:49.352987 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:49.352943    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 13:49:49.353697 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:49.353665    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 13:49:50.352803 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:50.352751    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 13:49:50.353045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:50.352832    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 13:49:51.356079 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:51.356042    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:51.356554 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:51.356301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:51.357175 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:51.357157    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 13:49:54.312242 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.312207    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"]
Sep 08 13:49:54.316873 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.316841    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.319857 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.319831    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-905fec-openshift-default-dockercfg-5x94q\""
Sep 08 13:49:54.331399 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.331342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"]
Sep 08 13:49:54.509919 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.509874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-data\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.509919 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.509920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lclb\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-kube-api-access-6lclb\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.510157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.509942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-podinfo\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.510157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.509971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-certs\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.510157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.509999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-token\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.510157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.510050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-socket\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.510157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.510108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee51e98-fe87-4857-a6bf-9094afefede5-istiod-ca-cert\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.510157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.510135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-credential-socket\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.510157 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.510149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-envoy\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.610661 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.610557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-token\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.610661 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.610623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-socket\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.610661 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.610657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee51e98-fe87-4857-a6bf-9094afefede5-istiod-ca-cert\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.610971 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.610687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-credential-socket\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.610971 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.610711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-envoy\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.610971 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.610743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-data\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.611130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.610997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lclb\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-kube-api-access-6lclb\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.611130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.611032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-podinfo\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.611130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.611076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-certs\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.611130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.611090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-socket\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.611130 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.611106    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-data\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.611404 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.611181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-credential-socket\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.611404 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.611350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-certs\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.611927 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.611903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee51e98-fe87-4857-a6bf-9094afefede5-istiod-ca-cert\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.613451 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.613423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-envoy\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.613582 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.613568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-podinfo\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.619543 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.619516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-token\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.619706 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.619684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lclb\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-kube-api-access-6lclb\") pod \"e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") " pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.626353 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.626328    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:54.904524 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.904496    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"]
Sep 08 13:49:54.906870 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:49:54.906842    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee51e98_fe87_4857_a6bf_9094afefede5.slice/crio-5d1914f2da68bf00db40bbe152a011669158938e4c2c96c0a6ecbc9aabe53be0 WatchSource:0}: Error finding container 5d1914f2da68bf00db40bbe152a011669158938e4c2c96c0a6ecbc9aabe53be0: Status 404 returned error can't find the container with id 5d1914f2da68bf00db40bbe152a011669158938e4c2c96c0a6ecbc9aabe53be0
Sep 08 13:49:54.909601 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.909570    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:54.909669 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.909640    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:54.909709 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:54.909677    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:49:55.816519 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:55.816475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" event={"ID":"7ee51e98-fe87-4857-a6bf-9094afefede5","Type":"ContainerStarted","Data":"35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522"}
Sep 08 13:49:55.817011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:55.816548    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" event={"ID":"7ee51e98-fe87-4857-a6bf-9094afefede5","Type":"ContainerStarted","Data":"5d1914f2da68bf00db40bbe152a011669158938e4c2c96c0a6ecbc9aabe53be0"}
Sep 08 13:49:56.627161 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:56.627110    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:56.632515 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:56.632483    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:56.657784 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:56.657727    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podStartSLOduration=2.657709024 podStartE2EDuration="2.657709024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:49:55.839327451 +0000 UTC m=+1304.190313411" watchObservedRunningTime="2026-09-08 13:49:56.657709024 +0000 UTC m=+1305.008694971"
Sep 08 13:49:56.768518 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:56.768487    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:49:56.822655 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:56.822617    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:56.823756 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:56.823726    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:49:59.795652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.795615    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"]
Sep 08 13:49:59.799186 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.799162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:49:59.802336 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.802311    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-905fec-dockercfg-48mqn\""
Sep 08 13:49:59.802501 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.802341    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-905fec\""
Sep 08 13:49:59.813772 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.813743    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"]
Sep 08 13:49:59.857224 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.857177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce9cece-477a-4b17-92cd-dd8a1869d000-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w\" (UID: \"7ce9cece-477a-4b17-92cd-dd8a1869d000\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:49:59.857400 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.857330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfmr2\" (UniqueName: \"kubernetes.io/projected/7ce9cece-477a-4b17-92cd-dd8a1869d000-kube-api-access-qfmr2\") pod \"payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w\" (UID: \"7ce9cece-477a-4b17-92cd-dd8a1869d000\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:49:59.958703 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.958658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfmr2\" (UniqueName: \"kubernetes.io/projected/7ce9cece-477a-4b17-92cd-dd8a1869d000-kube-api-access-qfmr2\") pod \"payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w\" (UID: \"7ce9cece-477a-4b17-92cd-dd8a1869d000\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:49:59.958912 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.958743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce9cece-477a-4b17-92cd-dd8a1869d000-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w\" (UID: \"7ce9cece-477a-4b17-92cd-dd8a1869d000\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:49:59.959534 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.959513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce9cece-477a-4b17-92cd-dd8a1869d000-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w\" (UID: \"7ce9cece-477a-4b17-92cd-dd8a1869d000\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:49:59.969445 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:49:59.969415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfmr2\" (UniqueName: \"kubernetes.io/projected/7ce9cece-477a-4b17-92cd-dd8a1869d000-kube-api-access-qfmr2\") pod \"payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w\" (UID: \"7ce9cece-477a-4b17-92cd-dd8a1869d000\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:50:00.109622 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:00.109522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:50:00.384141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:00.384109    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"]
Sep 08 13:50:00.386870 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:50:00.386836    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce9cece_477a_4b17_92cd_dd8a1869d000.slice/crio-9ec1c3b0a04949ae41f19e668458211d0eed3bb7f3c3d718f9ada3de1437f906 WatchSource:0}: Error finding container 9ec1c3b0a04949ae41f19e668458211d0eed3bb7f3c3d718f9ada3de1437f906: Status 404 returned error can't find the container with id 9ec1c3b0a04949ae41f19e668458211d0eed3bb7f3c3d718f9ada3de1437f906
Sep 08 13:50:00.846883 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:00.846840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w" event={"ID":"7ce9cece-477a-4b17-92cd-dd8a1869d000","Type":"ContainerStarted","Data":"2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2"}
Sep 08 13:50:00.847428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:00.846946    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:50:00.847428 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:00.846969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w" event={"ID":"7ce9cece-477a-4b17-92cd-dd8a1869d000","Type":"ContainerStarted","Data":"9ec1c3b0a04949ae41f19e668458211d0eed3bb7f3c3d718f9ada3de1437f906"}
Sep 08 13:50:00.864723 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:00.864664    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w" podStartSLOduration=1.86464669 podStartE2EDuration="1.86464669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:00.863492829 +0000 UTC m=+1309.214478776" watchObservedRunningTime="2026-09-08 13:50:00.86464669 +0000 UTC m=+1309.215632638"
Sep 08 13:50:07.666034 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.665997    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"]
Sep 08 13:50:07.670635 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.670612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.673305 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.673271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d9970e-openshift-default-dockercfg-49xxd\""
Sep 08 13:50:07.684045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.684015    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"]
Sep 08 13:50:07.730262 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-token\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.730262 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-podinfo\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.730526 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1575e4ae-d4c0-4007-adcc-29b52a10d532-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.730526 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-socket\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.730526 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-certs\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.730526 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-credential-socket\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.730664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-envoy\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.730664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-data\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.730664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.730634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7dc\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-kube-api-access-zd7dc\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.831966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1575e4ae-d4c0-4007-adcc-29b52a10d532-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832011 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-socket\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-certs\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-credential-socket\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-envoy\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-data\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7dc\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-kube-api-access-zd7dc\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-token\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832278 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-podinfo\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832642 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-socket\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832642 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-credential-socket\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832642 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-certs\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832805 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-data\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.832805 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.832784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1575e4ae-d4c0-4007-adcc-29b52a10d532-istiod-ca-cert\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.834628 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.834605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-envoy\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.838345 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.834815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-podinfo\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.842303 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.842272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-token\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.844817 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.844791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7dc\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-kube-api-access-zd7dc\") pod \"e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") " pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:07.982874 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:07.982832    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:08.285542 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:08.285515    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"]
Sep 08 13:50:08.287250 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:50:08.287220    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1575e4ae_d4c0_4007_adcc_29b52a10d532.slice/crio-dd964e86d80c7513a8c5b6961f23d7d4ff11f8b68ee46513e9c5562caeee8660 WatchSource:0}: Error finding container dd964e86d80c7513a8c5b6961f23d7d4ff11f8b68ee46513e9c5562caeee8660: Status 404 returned error can't find the container with id dd964e86d80c7513a8c5b6961f23d7d4ff11f8b68ee46513e9c5562caeee8660
Sep 08 13:50:08.289388 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:08.289348    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:08.289458 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:08.289427    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:08.289497 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:08.289460    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:08.889018 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:08.888981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" event={"ID":"1575e4ae-d4c0-4007-adcc-29b52a10d532","Type":"ContainerStarted","Data":"f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927"}
Sep 08 13:50:08.889448 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:08.889033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" event={"ID":"1575e4ae-d4c0-4007-adcc-29b52a10d532","Type":"ContainerStarted","Data":"dd964e86d80c7513a8c5b6961f23d7d4ff11f8b68ee46513e9c5562caeee8660"}
Sep 08 13:50:08.914173 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:08.914124    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" podStartSLOduration=1.914108541 podStartE2EDuration="1.914108541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:08.912766831 +0000 UTC m=+1317.263752789" watchObservedRunningTime="2026-09-08 13:50:08.914108541 +0000 UTC m=+1317.265094547"
Sep 08 13:50:08.983858 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:08.983819    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:09.988172 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:09.988137    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:10.898959 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:10.898918    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:10.899933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:10.899908    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:11.853488 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:11.853409    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 13:50:12.302685 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.302649    2561 scope.go:117] "RemoveContainer" containerID="726da0d110b40b8a6503ad4f3a0c1d21b55481b81b5f6257f4592528edab8d29"
Sep 08 13:50:12.708839 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.708806    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"]
Sep 08 13:50:12.713610 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.713589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:12.716387 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.716348    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d9970e-dockercfg-d5vcc\""
Sep 08 13:50:12.716505 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.716367    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d9970e\""
Sep 08 13:50:12.723852 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.723746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"]
Sep 08 13:50:12.796762 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.796729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4qz\" (UniqueName: \"kubernetes.io/projected/86711db8-c910-4192-b453-fd6b6add55b7-kube-api-access-5f4qz\") pod \"payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r\" (UID: \"86711db8-c910-4192-b453-fd6b6add55b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:12.796984 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.796785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86711db8-c910-4192-b453-fd6b6add55b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r\" (UID: \"86711db8-c910-4192-b453-fd6b6add55b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:12.865611 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.865560    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 13:50:12.865996 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.865629    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 13:50:12.898067 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.898033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86711db8-c910-4192-b453-fd6b6add55b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r\" (UID: \"86711db8-c910-4192-b453-fd6b6add55b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:12.898153 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.898110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4qz\" (UniqueName: \"kubernetes.io/projected/86711db8-c910-4192-b453-fd6b6add55b7-kube-api-access-5f4qz\") pod \"payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r\" (UID: \"86711db8-c910-4192-b453-fd6b6add55b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:12.899055 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.899031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86711db8-c910-4192-b453-fd6b6add55b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r\" (UID: \"86711db8-c910-4192-b453-fd6b6add55b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:12.907662 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:12.907633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4qz\" (UniqueName: \"kubernetes.io/projected/86711db8-c910-4192-b453-fd6b6add55b7-kube-api-access-5f4qz\") pod \"payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r\" (UID: \"86711db8-c910-4192-b453-fd6b6add55b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:13.027048 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:13.026945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:13.300441 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:13.300410    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"]
Sep 08 13:50:13.301516 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:50:13.301483    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86711db8_c910_4192_b453_fd6b6add55b7.slice/crio-3005c185cdbe11a84e638fd11ab1597389480626d2dcb8355f27e01f8d1e8878 WatchSource:0}: Error finding container 3005c185cdbe11a84e638fd11ab1597389480626d2dcb8355f27e01f8d1e8878: Status 404 returned error can't find the container with id 3005c185cdbe11a84e638fd11ab1597389480626d2dcb8355f27e01f8d1e8878
Sep 08 13:50:13.916265 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:13.916220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r" event={"ID":"86711db8-c910-4192-b453-fd6b6add55b7","Type":"ContainerStarted","Data":"bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55"}
Sep 08 13:50:13.916703 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:13.916297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r" event={"ID":"86711db8-c910-4192-b453-fd6b6add55b7","Type":"ContainerStarted","Data":"3005c185cdbe11a84e638fd11ab1597389480626d2dcb8355f27e01f8d1e8878"}
Sep 08 13:50:13.916703 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:13.916341    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:13.934691 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:13.934640    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r" podStartSLOduration=1.934623839 podStartE2EDuration="1.934623839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:13.932799941 +0000 UTC m=+1322.283785889" watchObservedRunningTime="2026-09-08 13:50:13.934623839 +0000 UTC m=+1322.285609787"
Sep 08 13:50:14.922277 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:14.922241    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerID="35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522" exitCode=0
Sep 08 13:50:14.922780 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:14.922317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" event={"ID":"7ee51e98-fe87-4857-a6bf-9094afefede5","Type":"ContainerDied","Data":"35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522"}
Sep 08 13:50:14.922780 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:14.922769    2561 scope.go:117] "RemoveContainer" containerID="35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522"
Sep 08 13:50:14.923882 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:14.923854    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:14.923958 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:14.923940    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:14.923996 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:14.923982    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:15.930532 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:15.930501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" event={"ID":"7ee51e98-fe87-4857-a6bf-9094afefede5","Type":"ContainerStarted","Data":"b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15"}
Sep 08 13:50:16.627101 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:16.627064    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:50:16.627319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:16.627296    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 13:50:16.627402 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:16.627360    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 13:50:17.627506 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:17.627465    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 13:50:17.627902 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:17.627533    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 13:50:18.627191 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:18.627150    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 13:50:18.627419 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:18.627226    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 13:50:19.627834 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:19.627794    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 13:50:19.628228 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:19.627856    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 13:50:20.627701 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:20.627662    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 13:50:20.627894 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:20.627745    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 13:50:21.627504 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:21.627464    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 13:50:21.627834 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:21.627792    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 13:50:22.630617 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:22.630586    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:50:22.631013 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:22.630948    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:50:22.631786 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:22.631767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 13:50:24.925312 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:24.925267    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:50:26.923313 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:26.922669    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 13:50:26.923313 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:26.922738    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 13:50:27.001866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:27.001831    2561 generic.go:356] "Generic (PLEG): container finished" podID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerID="f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927" exitCode=0
Sep 08 13:50:27.002029 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:27.001901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" event={"ID":"1575e4ae-d4c0-4007-adcc-29b52a10d532","Type":"ContainerDied","Data":"f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927"}
Sep 08 13:50:27.002304 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:27.002291    2561 scope.go:117] "RemoveContainer" containerID="f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927"
Sep 08 13:50:27.003617 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:27.003585    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:27.003717 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:27.003656    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:27.003717 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:27.003692    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:50:28.008288 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:28.008255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" event={"ID":"1575e4ae-d4c0-4007-adcc-29b52a10d532","Type":"ContainerStarted","Data":"f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb"}
Sep 08 13:50:28.983371 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:28.983324    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:28.983654 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:28.983625    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 08 13:50:28.983703 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:28.983684    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 08 13:50:29.983740 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:29.983700    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 08 13:50:29.984139 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:29.983760    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 08 13:50:30.983438 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:30.983369    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 08 13:50:30.983628 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:30.983454    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 08 13:50:31.984246 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:31.984208    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 08 13:50:31.984674 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:31.984281    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 08 13:50:32.984081 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:32.984043    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 08 13:50:32.984265 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:32.984106    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 08 13:50:33.987090 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:33.987059    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:33.987604 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:33.987408    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:33.988138 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:33.988123    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:50:36.443972 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.443894    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"]
Sep 08 13:50:36.448040 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.448023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.452570 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.452545    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1f082e\"/\"auth-test-model-1f082e-kserve-self-signed-certs\""
Sep 08 13:50:36.452570 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.452566    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1f082e\"/\"openshift-service-ca.crt\""
Sep 08 13:50:36.452753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.452546    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1f082e\"/\"kube-root-ca.crt\""
Sep 08 13:50:36.452753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.452545    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1f082e\"/\"default-dockercfg-22wdv\""
Sep 08 13:50:36.458684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.458663    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"]
Sep 08 13:50:36.519559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.519508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-home\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.519753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.519594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tmp-dir\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.519753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.519638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-model-cache\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.519753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.519674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-dshm\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.519753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.519712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq2b9\" (UniqueName: \"kubernetes.io/projected/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-kube-api-access-zq2b9\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.519753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.519739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tls-certs\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.620421 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.620355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq2b9\" (UniqueName: \"kubernetes.io/projected/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-kube-api-access-zq2b9\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.620421 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.620418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tls-certs\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.620684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.620482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-home\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.620684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.620527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tmp-dir\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.620684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.620564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-model-cache\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.620684 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.620599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-dshm\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.621021 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.620993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-model-cache\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.621095 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.621060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-home\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.621095 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.621082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tmp-dir\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.622808 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.622786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-dshm\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.623224 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.623202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tls-certs\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.630039 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.630013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq2b9\" (UniqueName: \"kubernetes.io/projected/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-kube-api-access-zq2b9\") pod \"auth-test-model-1f082e-kserve-7f87f76bf9-76nz9\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:36.758159 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:36.758073    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:37.034318 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:37.034221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"]
Sep 08 13:50:37.036914 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:50:37.036876    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode52f4c8f_1c05_40e0_8ffa_d640ac4385aa.slice/crio-d1b44a553183c74e04e36b777f5a41e191d6341ef18b38aa2a7fc4c2b574b1a7 WatchSource:0}: Error finding container d1b44a553183c74e04e36b777f5a41e191d6341ef18b38aa2a7fc4c2b574b1a7: Status 404 returned error can't find the container with id d1b44a553183c74e04e36b777f5a41e191d6341ef18b38aa2a7fc4c2b574b1a7
Sep 08 13:50:37.057681 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:37.057653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9" event={"ID":"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa","Type":"ContainerStarted","Data":"d1b44a553183c74e04e36b777f5a41e191d6341ef18b38aa2a7fc4c2b574b1a7"}
Sep 08 13:50:38.064725 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:38.064682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9" event={"ID":"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa","Type":"ContainerStarted","Data":"de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095"}
Sep 08 13:50:38.081564 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:38.081514    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9" podStartSLOduration=2.081497013 podStartE2EDuration="2.081497013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:38.081490577 +0000 UTC m=+1346.432476578" watchObservedRunningTime="2026-09-08 13:50:38.081497013 +0000 UTC m=+1346.432482961"
Sep 08 13:50:46.758491 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:46.758434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:46.758491 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:46.758504    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:46.772086 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:46.772059    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:50:47.122404 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:50:47.122297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:51:12.334585 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:12.334554    2561 scope.go:117] "RemoveContainer" containerID="e38f9adfd0ed38c47fb59f71a0550835afa6244da6ed7bda7f5e8ae4ce1f2a0e"
Sep 08 13:51:12.342666 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:12.342647    2561 scope.go:117] "RemoveContainer" containerID="91670510832b9d516d1a7b7db99f44d1f692d8b86ddee28d10477d2e5bb01f46"
Sep 08 13:51:12.349908 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:12.349877    2561 scope.go:117] "RemoveContainer" containerID="61c491a0494023e87010f85511e3dd649b1b6156c33b6677fbfef760cd82abe1"
Sep 08 13:51:17.665709 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.665674    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"]
Sep 08 13:51:17.670421 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.670400    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.673423 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.673399    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-905fec\"/\"openshift-service-ca.crt\""
Sep 08 13:51:17.673578 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.673559    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-905fec\"/\"kube-root-ca.crt\""
Sep 08 13:51:17.675085 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.675068    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-905fec\"/\"default-dockercfg-fxtd9\""
Sep 08 13:51:17.675158 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.675105    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-905fec\"/\"auth-test-model-905fec-kserve-self-signed-certs\""
Sep 08 13:51:17.681789 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.681764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"]
Sep 08 13:51:17.780892 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.780856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-dshm\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.781066 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.780948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-tmp-dir\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.781066 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.780973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6260391d-ce19-4877-85f6-a0fae0a7d582-tls-certs\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.781066 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.781028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-model-cache\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.781066 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.781045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-home\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.781206 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.781113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpdxm\" (UniqueName: \"kubernetes.io/projected/6260391d-ce19-4877-85f6-a0fae0a7d582-kube-api-access-xpdxm\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.882473 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-tmp-dir\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.882473 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6260391d-ce19-4877-85f6-a0fae0a7d582-tls-certs\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.882743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-model-cache\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.882743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-home\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.882743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpdxm\" (UniqueName: \"kubernetes.io/projected/6260391d-ce19-4877-85f6-a0fae0a7d582-kube-api-access-xpdxm\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.882743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-dshm\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.882968 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-tmp-dir\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.882968 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-home\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.883047 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.882981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-model-cache\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.884839 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.884823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-dshm\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.885105 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.885088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6260391d-ce19-4877-85f6-a0fae0a7d582-tls-certs\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.891946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.891926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpdxm\" (UniqueName: \"kubernetes.io/projected/6260391d-ce19-4877-85f6-a0fae0a7d582-kube-api-access-xpdxm\") pod \"auth-test-model-905fec-kserve-7f6b76f499-g9tsx\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") " pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:17.979855 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:17.979815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:18.252097 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:18.252075    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"]
Sep 08 13:51:18.254038 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:51:18.254010    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6260391d_ce19_4877_85f6_a0fae0a7d582.slice/crio-897830bc60141c0769bc9f9f837c81d2fae7932936c7099c9de20dae0a88432e WatchSource:0}: Error finding container 897830bc60141c0769bc9f9f837c81d2fae7932936c7099c9de20dae0a88432e: Status 404 returned error can't find the container with id 897830bc60141c0769bc9f9f837c81d2fae7932936c7099c9de20dae0a88432e
Sep 08 13:51:18.276206 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:18.276180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx" event={"ID":"6260391d-ce19-4877-85f6-a0fae0a7d582","Type":"ContainerStarted","Data":"897830bc60141c0769bc9f9f837c81d2fae7932936c7099c9de20dae0a88432e"}
Sep 08 13:51:19.282708 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:19.282673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx" event={"ID":"6260391d-ce19-4877-85f6-a0fae0a7d582","Type":"ContainerStarted","Data":"83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a"}
Sep 08 13:51:19.301509 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:19.301457    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx" podStartSLOduration=2.301440236 podStartE2EDuration="2.301440236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:51:19.299893555 +0000 UTC m=+1387.650879504" watchObservedRunningTime="2026-09-08 13:51:19.301440236 +0000 UTC m=+1387.652426183"
Sep 08 13:51:27.980658 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:27.980610    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:27.980658 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:27.980669    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:27.993293 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:27.993264    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:51:28.342260 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:51:28.342166    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:52:12.375076 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:52:12.374992    2561 scope.go:117] "RemoveContainer" containerID="6f3d8f4ea0e26a75b7483b5cf8d6a3e7f361f606ceb4c9f206640a0b3a1ea8aa"
Sep 08 13:52:12.383896 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:52:12.383876    2561 scope.go:117] "RemoveContainer" containerID="a06735bea3f1c8c992b9fb428a3580be00513681471b16a6b26a15c50e50b848"
Sep 08 13:52:12.391154 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:52:12.391135    2561 scope.go:117] "RemoveContainer" containerID="ab34a5f0c973415a35bbe24df9253fb53b5b9261c909d8b3ed79a2b378e401ef"
Sep 08 13:52:12.400060 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:52:12.399883    2561 scope.go:117] "RemoveContainer" containerID="e4f94232d9676c91bbf0b5ab936bb687c7afc94d5c7ecf330723520c1b9001ab"
Sep 08 13:53:25.317939 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:25.317902    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"]
Sep 08 13:53:26.432215 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:26.432180    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"]
Sep 08 13:53:40.337492 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.337443    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9" podUID="e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" containerName="main" containerID="cri-o://de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095" gracePeriod=45
Sep 08 13:53:40.582205 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.582178    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:53:40.734238 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734206    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq2b9\" (UniqueName: \"kubernetes.io/projected/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-kube-api-access-zq2b9\") pod \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") "
Sep 08 13:53:40.734451 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734251    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tmp-dir\") pod \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") "
Sep 08 13:53:40.734451 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tls-certs\") pod \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") "
Sep 08 13:53:40.734451 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734351    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-dshm\") pod \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") "
Sep 08 13:53:40.734451 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-home\") pod \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") "
Sep 08 13:53:40.734688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-model-cache\") pod \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\" (UID: \"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa\") "
Sep 08 13:53:40.734688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734654    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-home" (OuterVolumeSpecName: "home") pod "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" (UID: "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:40.734688 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734669    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" (UID: "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:40.734837 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734768    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" (UID: "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:40.734896 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734883    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:40.734949 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734904    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:40.734949 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.734921    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:40.736629 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.736607    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" (UID: "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:53:40.736710 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.736646    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-dshm" (OuterVolumeSpecName: "dshm") pod "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" (UID: "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:40.736787 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.736708    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-kube-api-access-zq2b9" (OuterVolumeSpecName: "kube-api-access-zq2b9") pod "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" (UID: "e52f4c8f-1c05-40e0-8ffa-d640ac4385aa"). InnerVolumeSpecName "kube-api-access-zq2b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:53:40.836280 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.836247    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:40.836280 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.836276    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:40.836280 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.836285    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq2b9\" (UniqueName: \"kubernetes.io/projected/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa-kube-api-access-zq2b9\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:40.966113 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.966072    2561 generic.go:356] "Generic (PLEG): container finished" podID="e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" containerID="de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095" exitCode=0
Sep 08 13:53:40.966286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.966136    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"
Sep 08 13:53:40.966286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.966157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9" event={"ID":"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa","Type":"ContainerDied","Data":"de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095"}
Sep 08 13:53:40.966286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.966204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9" event={"ID":"e52f4c8f-1c05-40e0-8ffa-d640ac4385aa","Type":"ContainerDied","Data":"d1b44a553183c74e04e36b777f5a41e191d6341ef18b38aa2a7fc4c2b574b1a7"}
Sep 08 13:53:40.966286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.966221    2561 scope.go:117] "RemoveContainer" containerID="de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095"
Sep 08 13:53:40.975022 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.975002    2561 scope.go:117] "RemoveContainer" containerID="de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095"
Sep 08 13:53:40.975276 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:53:40.975259    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095\": container with ID starting with de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095 not found: ID does not exist" containerID="de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095"
Sep 08 13:53:40.975322 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.975284    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095"} err="failed to get container status \"de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095\": rpc error: code = NotFound desc = could not find container \"de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095\": container with ID starting with de76fd3b40b939d627f38dab975a20694817ca5283b90506bf9d6165456de095 not found: ID does not exist"
Sep 08 13:53:40.988736 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:40.988682    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"]
Sep 08 13:53:41.001891 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.001866    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/auth-test-model-1f082e-kserve-7f87f76bf9-76nz9"]
Sep 08 13:53:41.451013 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.450975    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx" podUID="6260391d-ce19-4877-85f6-a0fae0a7d582" containerName="main" containerID="cri-o://83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a" gracePeriod=45
Sep 08 13:53:41.701132 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.701080    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:53:41.845027 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.844989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-home\") pod \"6260391d-ce19-4877-85f6-a0fae0a7d582\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") "
Sep 08 13:53:41.845199 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-dshm\") pod \"6260391d-ce19-4877-85f6-a0fae0a7d582\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") "
Sep 08 13:53:41.845199 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-tmp-dir\") pod \"6260391d-ce19-4877-85f6-a0fae0a7d582\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") "
Sep 08 13:53:41.845199 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-model-cache\") pod \"6260391d-ce19-4877-85f6-a0fae0a7d582\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") "
Sep 08 13:53:41.845411 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6260391d-ce19-4877-85f6-a0fae0a7d582-tls-certs\") pod \"6260391d-ce19-4877-85f6-a0fae0a7d582\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") "
Sep 08 13:53:41.845411 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845218    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpdxm\" (UniqueName: \"kubernetes.io/projected/6260391d-ce19-4877-85f6-a0fae0a7d582-kube-api-access-xpdxm\") pod \"6260391d-ce19-4877-85f6-a0fae0a7d582\" (UID: \"6260391d-ce19-4877-85f6-a0fae0a7d582\") "
Sep 08 13:53:41.845411 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845280    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-home" (OuterVolumeSpecName: "home") pod "6260391d-ce19-4877-85f6-a0fae0a7d582" (UID: "6260391d-ce19-4877-85f6-a0fae0a7d582"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:41.845580 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845430    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6260391d-ce19-4877-85f6-a0fae0a7d582" (UID: "6260391d-ce19-4877-85f6-a0fae0a7d582"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:41.845580 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845531    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:41.845580 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845554    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:41.845687 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.845571    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-model-cache" (OuterVolumeSpecName: "model-cache") pod "6260391d-ce19-4877-85f6-a0fae0a7d582" (UID: "6260391d-ce19-4877-85f6-a0fae0a7d582"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:41.847501 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.847474    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-dshm" (OuterVolumeSpecName: "dshm") pod "6260391d-ce19-4877-85f6-a0fae0a7d582" (UID: "6260391d-ce19-4877-85f6-a0fae0a7d582"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:41.847501 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.847482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6260391d-ce19-4877-85f6-a0fae0a7d582-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6260391d-ce19-4877-85f6-a0fae0a7d582" (UID: "6260391d-ce19-4877-85f6-a0fae0a7d582"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:53:41.847637 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.847482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6260391d-ce19-4877-85f6-a0fae0a7d582-kube-api-access-xpdxm" (OuterVolumeSpecName: "kube-api-access-xpdxm") pod "6260391d-ce19-4877-85f6-a0fae0a7d582" (UID: "6260391d-ce19-4877-85f6-a0fae0a7d582"). InnerVolumeSpecName "kube-api-access-xpdxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:53:41.946414 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.946355    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:41.946414 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.946409    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6260391d-ce19-4877-85f6-a0fae0a7d582-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:41.946414 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.946420    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6260391d-ce19-4877-85f6-a0fae0a7d582-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:41.946647 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.946429    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpdxm\" (UniqueName: \"kubernetes.io/projected/6260391d-ce19-4877-85f6-a0fae0a7d582-kube-api-access-xpdxm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:53:41.973151 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.973076    2561 generic.go:356] "Generic (PLEG): container finished" podID="6260391d-ce19-4877-85f6-a0fae0a7d582" containerID="83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a" exitCode=0
Sep 08 13:53:41.973151 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.973116    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"
Sep 08 13:53:41.973151 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.973132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx" event={"ID":"6260391d-ce19-4877-85f6-a0fae0a7d582","Type":"ContainerDied","Data":"83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a"}
Sep 08 13:53:41.973338 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.973168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx" event={"ID":"6260391d-ce19-4877-85f6-a0fae0a7d582","Type":"ContainerDied","Data":"897830bc60141c0769bc9f9f837c81d2fae7932936c7099c9de20dae0a88432e"}
Sep 08 13:53:41.973338 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.973187    2561 scope.go:117] "RemoveContainer" containerID="83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a"
Sep 08 13:53:41.982567 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.982548    2561 scope.go:117] "RemoveContainer" containerID="83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a"
Sep 08 13:53:41.982825 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:53:41.982806    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a\": container with ID starting with 83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a not found: ID does not exist" containerID="83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a"
Sep 08 13:53:41.982897 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.982831    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a"} err="failed to get container status \"83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a\": rpc error: code = NotFound desc = could not find container \"83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a\": container with ID starting with 83a848e126d949f03d367a50240e5c01dc20085941533aed492ef7df7306082a not found: ID does not exist"
Sep 08 13:53:41.995711 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.995691    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"]
Sep 08 13:53:41.999574 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:41.999554    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/auth-test-model-905fec-kserve-7f6b76f499-g9tsx"]
Sep 08 13:53:42.245495 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:42.245419    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6260391d-ce19-4877-85f6-a0fae0a7d582" path="/var/lib/kubelet/pods/6260391d-ce19-4877-85f6-a0fae0a7d582/volumes"
Sep 08 13:53:42.245822 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:42.245810    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" path="/var/lib/kubelet/pods/e52f4c8f-1c05-40e0-8ffa-d640ac4385aa/volumes"
Sep 08 13:53:51.235232 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.235198    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"]
Sep 08 13:53:51.235826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.235534    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6260391d-ce19-4877-85f6-a0fae0a7d582" containerName="main"
Sep 08 13:53:51.235826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.235546    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6260391d-ce19-4877-85f6-a0fae0a7d582" containerName="main"
Sep 08 13:53:51.235826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.235564    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" containerName="main"
Sep 08 13:53:51.235826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.235569    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" containerName="main"
Sep 08 13:53:51.235826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.235618    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6260391d-ce19-4877-85f6-a0fae0a7d582" containerName="main"
Sep 08 13:53:51.235826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.235629    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e52f4c8f-1c05-40e0-8ffa-d640ac4385aa" containerName="main"
Sep 08 13:53:51.238543 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.238521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.242665 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.242645    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-905fec\"/\"openshift-service-ca.crt\""
Sep 08 13:53:51.242810 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.242786    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-905fec\"/\"default-dockercfg-fxtd9\""
Sep 08 13:53:51.242888 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.242797    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-905fec\"/\"sub-test-model-905fec-kserve-self-signed-certs\""
Sep 08 13:53:51.242888 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.242797    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-905fec\"/\"kube-root-ca.crt\""
Sep 08 13:53:51.250811 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.250785    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"]
Sep 08 13:53:51.327813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.327781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-dshm\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.327986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.327849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-model-cache\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.327986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.327868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzn4\" (UniqueName: \"kubernetes.io/projected/4f2a7436-5e4e-4cf0-9540-422c9727f46c-kube-api-access-rvzn4\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.327986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.327909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tls-certs\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.327986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.327969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-home\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.328152 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.328025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tmp-dir\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.428652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.428612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tmp-dir\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.428652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.428655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-dshm\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.428903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.428692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-model-cache\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.428903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.428710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzn4\" (UniqueName: \"kubernetes.io/projected/4f2a7436-5e4e-4cf0-9540-422c9727f46c-kube-api-access-rvzn4\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.428903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.428737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tls-certs\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.428903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.428764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-home\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.429126 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.429040    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tmp-dir\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.429126 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.429091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-model-cache\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.429230 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.429135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-home\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.431039 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.431019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-dshm\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.431481 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.431460    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tls-certs\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.437445 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.437421    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzn4\" (UniqueName: \"kubernetes.io/projected/4f2a7436-5e4e-4cf0-9540-422c9727f46c-kube-api-access-rvzn4\") pod \"sub-test-model-905fec-kserve-744655f87f-6k5gr\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") " pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.553780 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.553693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:53:51.825523 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:51.825495    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"]
Sep 08 13:53:51.827327 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:53:51.827302    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f2a7436_5e4e_4cf0_9540_422c9727f46c.slice/crio-537518a48c7ab7713679ab2c7b3311db7834f06cc5dd1d164c3556bbe8672f15 WatchSource:0}: Error finding container 537518a48c7ab7713679ab2c7b3311db7834f06cc5dd1d164c3556bbe8672f15: Status 404 returned error can't find the container with id 537518a48c7ab7713679ab2c7b3311db7834f06cc5dd1d164c3556bbe8672f15
Sep 08 13:53:52.021193 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:52.021158    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr" event={"ID":"4f2a7436-5e4e-4cf0-9540-422c9727f46c","Type":"ContainerStarted","Data":"93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a"}
Sep 08 13:53:52.021408 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:52.021203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr" event={"ID":"4f2a7436-5e4e-4cf0-9540-422c9727f46c","Type":"ContainerStarted","Data":"537518a48c7ab7713679ab2c7b3311db7834f06cc5dd1d164c3556bbe8672f15"}
Sep 08 13:53:52.041564 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:53:52.041512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr" podStartSLOduration=1.041492017 podStartE2EDuration="1.041492017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:53:52.040922337 +0000 UTC m=+1540.391908281" watchObservedRunningTime="2026-09-08 13:53:52.041492017 +0000 UTC m=+1540.392477966"
Sep 08 13:54:01.553815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:01.553777    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:54:01.553815 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:01.553827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:54:01.566438 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:01.566418    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:54:02.077820 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:02.077789    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:54:12.433883 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:12.433851    2561 scope.go:117] "RemoveContainer" containerID="368dd458bde5dbd84dbdd51efe96a39a0da4dd1ec40a596b8c016b1f68ae86e2"
Sep 08 13:54:44.369953 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.369919    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"]
Sep 08 13:54:44.372008 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.371991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.374925 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.374901    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8ef041-openshift-default-dockercfg-qnl6k\""
Sep 08 13:54:44.386946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.386922    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"]
Sep 08 13:54:44.399881 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.399855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-data\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.399995 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.399894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-socket\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.399995 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.399924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtg6b\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-kube-api-access-mtg6b\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.400122 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.400021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-envoy\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.400122 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.400082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-token\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.400232 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.400126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-podinfo\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.400232 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.400182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-certs\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.400232 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.400215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-credential-socket\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.400399 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.400257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istiod-ca-cert\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-credential-socket\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501286 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istiod-ca-cert\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-data\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-socket\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtg6b\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-kube-api-access-mtg6b\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-envoy\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-token\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-podinfo\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501523    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-certs\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501734    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-credential-socket\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.501930 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.501792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-socket\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.502045 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.502017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-certs\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.502100 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.502052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-data\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.502142 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.502093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istiod-ca-cert\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.503918 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.503899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-envoy\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.504012 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.503995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-podinfo\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.509453 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.509431    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-token\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.509544 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.509525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtg6b\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-kube-api-access-mtg6b\") pod \"e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") " pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.684090 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.684058    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:44.981521 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.981491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"]
Sep 08 13:54:44.982821 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:54:44.982789    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150c9d02_eb92_4e86_a874_d45b4cddb7b2.slice/crio-7922490843d04cbcf8b473c024ad1a705f0780d3f7bb15280a087c106a2b3b12 WatchSource:0}: Error finding container 7922490843d04cbcf8b473c024ad1a705f0780d3f7bb15280a087c106a2b3b12: Status 404 returned error can't find the container with id 7922490843d04cbcf8b473c024ad1a705f0780d3f7bb15280a087c106a2b3b12
Sep 08 13:54:44.985167 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.985130    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:54:44.985289 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.985211    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:54:44.985289 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.985254    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:54:44.990526 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:44.990507    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"]
Sep 08 13:54:45.268826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:45.268742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" event={"ID":"150c9d02-eb92-4e86-a874-d45b4cddb7b2","Type":"ContainerStarted","Data":"b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793"}
Sep 08 13:54:45.268826 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:45.268800    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" event={"ID":"150c9d02-eb92-4e86-a874-d45b4cddb7b2","Type":"ContainerStarted","Data":"7922490843d04cbcf8b473c024ad1a705f0780d3f7bb15280a087c106a2b3b12"}
Sep 08 13:54:45.289526 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:45.289471    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" podStartSLOduration=1.289455454 podStartE2EDuration="1.289455454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:54:45.288309513 +0000 UTC m=+1593.639295460" watchObservedRunningTime="2026-09-08 13:54:45.289455454 +0000 UTC m=+1593.640441402"
Sep 08 13:54:45.684982 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:45.684949    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:46.689917 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:46.689885    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:47.278411 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:47.278352    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:47.279456 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:47.279434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:54:49.049232 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.049194    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"]
Sep 08 13:54:49.053143 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.053117    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:49.056489 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.056451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8ef041-dockercfg-94fj8\""
Sep 08 13:54:49.056605 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.056470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8ef041\""
Sep 08 13:54:49.061144 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.061120    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"]
Sep 08 13:54:49.142812 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.142768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8ef041-64cd964477-m297q\" (UID: \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:49.142988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.142841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7b7h\" (UniqueName: \"kubernetes.io/projected/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-kube-api-access-s7b7h\") pod \"payload-processing-e2e-api-b-8ef041-64cd964477-m297q\" (UID: \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:49.244010 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.243971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8ef041-64cd964477-m297q\" (UID: \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:49.244193 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.244045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7b7h\" (UniqueName: \"kubernetes.io/projected/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-kube-api-access-s7b7h\") pod \"payload-processing-e2e-api-b-8ef041-64cd964477-m297q\" (UID: \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:49.244872 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.244851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8ef041-64cd964477-m297q\" (UID: \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:49.252434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.252414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7b7h\" (UniqueName: \"kubernetes.io/projected/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-kube-api-access-s7b7h\") pod \"payload-processing-e2e-api-b-8ef041-64cd964477-m297q\" (UID: \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\") " pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:49.366728 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.366646    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:49.636235 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:49.636161    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"]
Sep 08 13:54:49.639371 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:54:49.639345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f5ad4d_b900_4e98_b7ee_1067883d1ada.slice/crio-41b223c2294e539f54773e7c57c205d20ac78a163b706efefa0e92293d8aefea WatchSource:0}: Error finding container 41b223c2294e539f54773e7c57c205d20ac78a163b706efefa0e92293d8aefea: Status 404 returned error can't find the container with id 41b223c2294e539f54773e7c57c205d20ac78a163b706efefa0e92293d8aefea
Sep 08 13:54:50.293724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:50.293690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q" event={"ID":"f5f5ad4d-b900-4e98-b7ee-1067883d1ada","Type":"ContainerStarted","Data":"c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965"}
Sep 08 13:54:50.294221 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:50.293745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q" event={"ID":"f5f5ad4d-b900-4e98-b7ee-1067883d1ada","Type":"ContainerStarted","Data":"41b223c2294e539f54773e7c57c205d20ac78a163b706efefa0e92293d8aefea"}
Sep 08 13:54:50.294221 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:50.293836    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:54:50.312414 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:50.312347    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q" podStartSLOduration=1.312326746 podStartE2EDuration="1.312326746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:54:50.310280155 +0000 UTC m=+1598.661266105" watchObservedRunningTime="2026-09-08 13:54:50.312326746 +0000 UTC m=+1598.663312695"
Sep 08 13:54:55.639922 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.639882    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"]
Sep 08 13:54:55.642800 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.642782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.646549 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.646524    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1f082e\"/\"openshift-service-ca.crt\""
Sep 08 13:54:55.646549 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.646539    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1f082e\"/\"kube-root-ca.crt\""
Sep 08 13:54:55.646732 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.646550    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1f082e\"/\"rate-test-model-1f082e-kserve-self-signed-certs\""
Sep 08 13:54:55.646732 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.646638    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1f082e\"/\"default-dockercfg-22wdv\""
Sep 08 13:54:55.651073 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.651049    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"]
Sep 08 13:54:55.700353 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.700326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937d2191-1010-4ae7-8454-84579de1a0be-tls-certs\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.700525 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.700363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-dshm\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.700525 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.700449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-tmp-dir\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.700525 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.700483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-model-cache\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.700649 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.700564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbrw\" (UniqueName: \"kubernetes.io/projected/937d2191-1010-4ae7-8454-84579de1a0be-kube-api-access-sqbrw\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.700649 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.700599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-home\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.801910 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.801874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-dshm\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.801910 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.801914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-tmp-dir\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.802145 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.801933    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-model-cache\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.802145 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.801989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbrw\" (UniqueName: \"kubernetes.io/projected/937d2191-1010-4ae7-8454-84579de1a0be-kube-api-access-sqbrw\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.802145 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.802020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-home\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.802145 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.802049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937d2191-1010-4ae7-8454-84579de1a0be-tls-certs\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.802356 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.802296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-tmp-dir\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.802491 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.802472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-model-cache\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.802533 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.802506    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-home\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.804364 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.804341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-dshm\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.804593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.804578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937d2191-1010-4ae7-8454-84579de1a0be-tls-certs\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.809895 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.809874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbrw\" (UniqueName: \"kubernetes.io/projected/937d2191-1010-4ae7-8454-84579de1a0be-kube-api-access-sqbrw\") pod \"rate-test-model-1f082e-kserve-5675d44c5f-gfb6h\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") " pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:55.952669 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:55.952622    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:54:56.221336 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:56.221312    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"]
Sep 08 13:54:56.224021 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:54:56.223990    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod937d2191_1010_4ae7_8454_84579de1a0be.slice/crio-f0af2923067bfa77d4e9a29258b73862f68219b25071a4c4ef95996f4a72a9eb WatchSource:0}: Error finding container f0af2923067bfa77d4e9a29258b73862f68219b25071a4c4ef95996f4a72a9eb: Status 404 returned error can't find the container with id f0af2923067bfa77d4e9a29258b73862f68219b25071a4c4ef95996f4a72a9eb
Sep 08 13:54:56.326107 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:56.326075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h" event={"ID":"937d2191-1010-4ae7-8454-84579de1a0be","Type":"ContainerStarted","Data":"9ddfdd2d583f134614e975b0429802b36bd9c1063413f9b67627177778bf40bf"}
Sep 08 13:54:56.326107 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:56.326113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h" event={"ID":"937d2191-1010-4ae7-8454-84579de1a0be","Type":"ContainerStarted","Data":"f0af2923067bfa77d4e9a29258b73862f68219b25071a4c4ef95996f4a72a9eb"}
Sep 08 13:54:56.345903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:54:56.345842    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h" podStartSLOduration=1.345820609 podStartE2EDuration="1.345820609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:54:56.34399623 +0000 UTC m=+1604.694982175" watchObservedRunningTime="2026-09-08 13:54:56.345820609 +0000 UTC m=+1604.696806558"
Sep 08 13:55:00.010412 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.010317    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr" podUID="4f2a7436-5e4e-4cf0-9540-422c9727f46c" containerName="main" containerID="cri-o://93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a" gracePeriod=45
Sep 08 13:55:00.263568 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.263499    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:55:00.344366 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.344330    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvzn4\" (UniqueName: \"kubernetes.io/projected/4f2a7436-5e4e-4cf0-9540-422c9727f46c-kube-api-access-rvzn4\") pod \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") "
Sep 08 13:55:00.344559 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.344521    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-home\") pod \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") "
Sep 08 13:55:00.344633 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.344558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-model-cache\") pod \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") "
Sep 08 13:55:00.344633 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.344585    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tls-certs\") pod \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") "
Sep 08 13:55:00.344741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.344671    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-dshm\") pod \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") "
Sep 08 13:55:00.344741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.344697    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tmp-dir\") pod \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\" (UID: \"4f2a7436-5e4e-4cf0-9540-422c9727f46c\") "
Sep 08 13:55:00.344852 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.344801    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f2a7436-5e4e-4cf0-9540-422c9727f46c" (UID: "4f2a7436-5e4e-4cf0-9540-422c9727f46c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:55:00.344852 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.344823    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-home" (OuterVolumeSpecName: "home") pod "4f2a7436-5e4e-4cf0-9540-422c9727f46c" (UID: "4f2a7436-5e4e-4cf0-9540-422c9727f46c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:55:00.345056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.345033    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f2a7436-5e4e-4cf0-9540-422c9727f46c" (UID: "4f2a7436-5e4e-4cf0-9540-422c9727f46c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:55:00.345056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.345048    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:00.345227 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.345109    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:00.345560 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.345530    2561 generic.go:356] "Generic (PLEG): container finished" podID="4f2a7436-5e4e-4cf0-9540-422c9727f46c" containerID="93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a" exitCode=0
Sep 08 13:55:00.345694 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.345606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr" event={"ID":"4f2a7436-5e4e-4cf0-9540-422c9727f46c","Type":"ContainerDied","Data":"93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a"}
Sep 08 13:55:00.345763 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.345699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr" event={"ID":"4f2a7436-5e4e-4cf0-9540-422c9727f46c","Type":"ContainerDied","Data":"537518a48c7ab7713679ab2c7b3311db7834f06cc5dd1d164c3556bbe8672f15"}
Sep 08 13:55:00.345763 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.345734    2561 scope.go:117] "RemoveContainer" containerID="93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a"
Sep 08 13:55:00.346251 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.345927    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"
Sep 08 13:55:00.347156 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.347121    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-dshm" (OuterVolumeSpecName: "dshm") pod "4f2a7436-5e4e-4cf0-9540-422c9727f46c" (UID: "4f2a7436-5e4e-4cf0-9540-422c9727f46c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:55:00.347678 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.347652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2a7436-5e4e-4cf0-9540-422c9727f46c-kube-api-access-rvzn4" (OuterVolumeSpecName: "kube-api-access-rvzn4") pod "4f2a7436-5e4e-4cf0-9540-422c9727f46c" (UID: "4f2a7436-5e4e-4cf0-9540-422c9727f46c"). InnerVolumeSpecName "kube-api-access-rvzn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:55:00.348192 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.348165    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f2a7436-5e4e-4cf0-9540-422c9727f46c" (UID: "4f2a7436-5e4e-4cf0-9540-422c9727f46c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:55:00.385475 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.385040    2561 scope.go:117] "RemoveContainer" containerID="93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a"
Sep 08 13:55:00.389876 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:55:00.388358    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a\": container with ID starting with 93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a not found: ID does not exist" containerID="93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a"
Sep 08 13:55:00.389876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.388426    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a"} err="failed to get container status \"93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a\": rpc error: code = NotFound desc = could not find container \"93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a\": container with ID starting with 93206839531b77de117a08bf2125f658d410219531e90bb00098de687640ae9a not found: ID does not exist"
Sep 08 13:55:00.446463 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.446418    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:00.446463 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.446470    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:00.446707 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.446482    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzn4\" (UniqueName: \"kubernetes.io/projected/4f2a7436-5e4e-4cf0-9540-422c9727f46c-kube-api-access-rvzn4\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:00.446707 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.446492    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f2a7436-5e4e-4cf0-9540-422c9727f46c-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:00.672135 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.672102    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"]
Sep 08 13:55:00.679518 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:00.679487    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-905fec/sub-test-model-905fec-kserve-744655f87f-6k5gr"]
Sep 08 13:55:01.302224 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:01.302193    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:55:02.249836 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:02.249787    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f2a7436-5e4e-4cf0-9540-422c9727f46c" path="/var/lib/kubelet/pods/4f2a7436-5e4e-4cf0-9540-422c9727f46c/volumes"
Sep 08 13:55:05.953403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:05.953296    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:55:05.954359 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:05.954340    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:55:05.971779 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:05.971752    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:55:06.389325 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:06.389238    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:55:10.401139 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:10.401044    2561 generic.go:356] "Generic (PLEG): container finished" podID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerID="b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793" exitCode=0
Sep 08 13:55:10.401139 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:10.401115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" event={"ID":"150c9d02-eb92-4e86-a874-d45b4cddb7b2","Type":"ContainerDied","Data":"b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793"}
Sep 08 13:55:10.401596 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:10.401515    2561 scope.go:117] "RemoveContainer" containerID="b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793"
Sep 08 13:55:10.402652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:10.402623    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:55:10.402722 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:10.402706    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:55:10.402759 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:10.402741    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:55:11.408916 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:11.408883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" event={"ID":"150c9d02-eb92-4e86-a874-d45b4cddb7b2","Type":"ContainerStarted","Data":"e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7"}
Sep 08 13:55:11.684360 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:11.684323    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:55:11.684591 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:11.684568    2561 patch_prober.go:28] interesting pod/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 08 13:55:11.684644 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:11.684619    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 08 13:55:12.459629 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:12.459592    2561 scope.go:117] "RemoveContainer" containerID="37ce922be4bfaf12eaad533b032207d948da3d47e8d92af8a81d683ba348820a"
Sep 08 13:55:12.684460 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:12.684424    2561 patch_prober.go:28] interesting pod/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 08 13:55:12.684635 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:12.684488    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 08 13:55:13.684649 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:13.684607    2561 patch_prober.go:28] interesting pod/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 08 13:55:13.685079 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:13.684676    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 08 13:55:14.684783 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:14.684746    2561 patch_prober.go:28] interesting pod/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 08 13:55:14.685183 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:14.684802    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 08 13:55:15.308106 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.308072    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"]
Sep 08 13:55:15.308456 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.308441    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f2a7436-5e4e-4cf0-9540-422c9727f46c" containerName="main"
Sep 08 13:55:15.308456 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.308457    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f2a7436-5e4e-4cf0-9540-422c9727f46c" containerName="main"
Sep 08 13:55:15.308562 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.308529    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f2a7436-5e4e-4cf0-9540-422c9727f46c" containerName="main"
Sep 08 13:55:15.311313 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.311297    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"
Sep 08 13:55:15.314601 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.314580    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 13:55:15.315127 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.315110    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 13:55:15.326646 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.326616    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"]
Sep 08 13:55:15.387211 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.387174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd5e5348-253a-41d0-9907-d6fdae3ea4f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-569975-7sgd7\" (UID: \"dd5e5348-253a-41d0-9907-d6fdae3ea4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"
Sep 08 13:55:15.487785 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.487748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd5e5348-253a-41d0-9907-d6fdae3ea4f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-569975-7sgd7\" (UID: \"dd5e5348-253a-41d0-9907-d6fdae3ea4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"
Sep 08 13:55:15.488434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.488406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd5e5348-253a-41d0-9907-d6fdae3ea4f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-569975-7sgd7\" (UID: \"dd5e5348-253a-41d0-9907-d6fdae3ea4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"
Sep 08 13:55:15.620724 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.620635    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"
Sep 08 13:55:15.685549 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.685512    2561 patch_prober.go:28] interesting pod/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 08 13:55:15.685924 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.685582    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 08 13:55:15.915060 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:15.915038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"]
Sep 08 13:55:15.917018 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:55:15.916979    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5e5348_253a_41d0_9907_d6fdae3ea4f6.slice/crio-6c2f5c56ee87f5cfb0bf6656eeadcae67bcfbfa087c058ea299569f3bc6b7080 WatchSource:0}: Error finding container 6c2f5c56ee87f5cfb0bf6656eeadcae67bcfbfa087c058ea299569f3bc6b7080: Status 404 returned error can't find the container with id 6c2f5c56ee87f5cfb0bf6656eeadcae67bcfbfa087c058ea299569f3bc6b7080
Sep 08 13:55:16.435679 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:16.435643    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7" event={"ID":"dd5e5348-253a-41d0-9907-d6fdae3ea4f6","Type":"ContainerStarted","Data":"4e7bec401fc1a9b0c790e4b28700c96fbcca53ff68a7eff5ce8f265580158744"}
Sep 08 13:55:16.436037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:16.435699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7" event={"ID":"dd5e5348-253a-41d0-9907-d6fdae3ea4f6","Type":"ContainerStarted","Data":"6c2f5c56ee87f5cfb0bf6656eeadcae67bcfbfa087c058ea299569f3bc6b7080"}
Sep 08 13:55:16.451546 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:16.451493    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7" podStartSLOduration=1.451477522 podStartE2EDuration="1.451477522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:55:16.450769639 +0000 UTC m=+1624.801755589" watchObservedRunningTime="2026-09-08 13:55:16.451477522 +0000 UTC m=+1624.802463470"
Sep 08 13:55:16.689534 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:16.689498    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:55:16.690019 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:16.689889    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:55:16.690570 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:16.690545    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:55:17.441704 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:17.441668    2561 generic.go:356] "Generic (PLEG): container finished" podID="dd5e5348-253a-41d0-9907-d6fdae3ea4f6" containerID="4e7bec401fc1a9b0c790e4b28700c96fbcca53ff68a7eff5ce8f265580158744" exitCode=0
Sep 08 13:55:17.441888 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:17.441746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7" event={"ID":"dd5e5348-253a-41d0-9907-d6fdae3ea4f6","Type":"ContainerDied","Data":"4e7bec401fc1a9b0c790e4b28700c96fbcca53ff68a7eff5ce8f265580158744"}
Sep 08 13:55:18.577673 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:18.577645    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"
Sep 08 13:55:18.717936 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:18.717837    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd5e5348-253a-41d0-9907-d6fdae3ea4f6-maas-api-service-ca\") pod \"dd5e5348-253a-41d0-9907-d6fdae3ea4f6\" (UID: \"dd5e5348-253a-41d0-9907-d6fdae3ea4f6\") "
Sep 08 13:55:18.718167 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:18.718144    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5e5348-253a-41d0-9907-d6fdae3ea4f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd5e5348-253a-41d0-9907-d6fdae3ea4f6" (UID: "dd5e5348-253a-41d0-9907-d6fdae3ea4f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:55:18.819027 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:18.818978    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd5e5348-253a-41d0-9907-d6fdae3ea4f6-maas-api-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:19.451945 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:19.451907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7" event={"ID":"dd5e5348-253a-41d0-9907-d6fdae3ea4f6","Type":"ContainerDied","Data":"6c2f5c56ee87f5cfb0bf6656eeadcae67bcfbfa087c058ea299569f3bc6b7080"}
Sep 08 13:55:19.451945 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:19.451936    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"
Sep 08 13:55:19.452156 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:19.451968    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c2f5c56ee87f5cfb0bf6656eeadcae67bcfbfa087c058ea299569f3bc6b7080"
Sep 08 13:55:31.342809 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:31.342777    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"]
Sep 08 13:55:31.347397 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:31.346599    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-569975-7sgd7"]
Sep 08 13:55:32.245003 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:32.244967    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd5e5348-253a-41d0-9907-d6fdae3ea4f6" path="/var/lib/kubelet/pods/dd5e5348-253a-41d0-9907-d6fdae3ea4f6/volumes"
Sep 08 13:55:54.127300 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.127265    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"]
Sep 08 13:55:54.127942 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.127532    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q" podUID="f5f5ad4d-b900-4e98-b7ee-1067883d1ada" containerName="payload-processing" containerID="cri-o://c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965" gracePeriod=30
Sep 08 13:55:54.381501 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.381441    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:55:54.447767 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.447734    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7b7h\" (UniqueName: \"kubernetes.io/projected/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-kube-api-access-s7b7h\") pod \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\" (UID: \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\") "
Sep 08 13:55:54.447950 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.447818    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-plugins-config-volume\") pod \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\" (UID: \"f5f5ad4d-b900-4e98-b7ee-1067883d1ada\") "
Sep 08 13:55:54.448182 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.448157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5f5ad4d-b900-4e98-b7ee-1067883d1ada" (UID: "f5f5ad4d-b900-4e98-b7ee-1067883d1ada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:55:54.449813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.449789    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-kube-api-access-s7b7h" (OuterVolumeSpecName: "kube-api-access-s7b7h") pod "f5f5ad4d-b900-4e98-b7ee-1067883d1ada" (UID: "f5f5ad4d-b900-4e98-b7ee-1067883d1ada"). InnerVolumeSpecName "kube-api-access-s7b7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:55:54.548547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.548512    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7b7h\" (UniqueName: \"kubernetes.io/projected/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-kube-api-access-s7b7h\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:54.548547 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.548544    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f5ad4d-b900-4e98-b7ee-1067883d1ada-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:55:54.615774 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.615738    2561 generic.go:356] "Generic (PLEG): container finished" podID="f5f5ad4d-b900-4e98-b7ee-1067883d1ada" containerID="c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965" exitCode=0
Sep 08 13:55:54.615953 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.615781    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"
Sep 08 13:55:54.615953 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.615797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q" event={"ID":"f5f5ad4d-b900-4e98-b7ee-1067883d1ada","Type":"ContainerDied","Data":"c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965"}
Sep 08 13:55:54.615953 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.615840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q" event={"ID":"f5f5ad4d-b900-4e98-b7ee-1067883d1ada","Type":"ContainerDied","Data":"41b223c2294e539f54773e7c57c205d20ac78a163b706efefa0e92293d8aefea"}
Sep 08 13:55:54.615953 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.615860    2561 scope.go:117] "RemoveContainer" containerID="c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965"
Sep 08 13:55:54.627046 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.627022    2561 scope.go:117] "RemoveContainer" containerID="c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965"
Sep 08 13:55:54.627340 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:55:54.627319    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965\": container with ID starting with c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965 not found: ID does not exist" containerID="c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965"
Sep 08 13:55:54.627455 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.627347    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965"} err="failed to get container status \"c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965\": rpc error: code = NotFound desc = could not find container \"c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965\": container with ID starting with c33c108390e9499a69c53fdfb29f18e4c225e79abc5e9b8f71d61634675ec965 not found: ID does not exist"
Sep 08 13:55:54.642923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.642862    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"]
Sep 08 13:55:54.647214 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:54.647190    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8ef041-64cd964477-m297q"]
Sep 08 13:55:56.245346 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:55:56.245312    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f5ad4d-b900-4e98-b7ee-1067883d1ada" path="/var/lib/kubelet/pods/f5f5ad4d-b900-4e98-b7ee-1067883d1ada/volumes"
Sep 08 13:56:01.190871 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:01.190838    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"]
Sep 08 13:56:01.191272 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:01.191066    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy" containerID="cri-o://e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7" gracePeriod=30
Sep 08 13:56:01.692177 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:01.692132    2561 patch_prober.go:28] interesting pod/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 13:56:01.692354 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:01.692190    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:56:06.451648 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.451618    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:56:06.556514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556439    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istiod-ca-cert\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.556514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556484    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-podinfo\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.556514 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556507    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-credential-socket\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.556988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-token\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.556988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556557    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtg6b\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-kube-api-access-mtg6b\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.556988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556582    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-data\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.556988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556596    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-certs\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.556988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556839    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:56:06.556988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:06.556988 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556982    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:06.557341 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:06.557341 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.556998    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-envoy\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.557341 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.557069    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-socket\") pod \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\" (UID: \"150c9d02-eb92-4e86-a874-d45b4cddb7b2\") "
Sep 08 13:56:06.557341 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.557319    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:06.557583 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.557494    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.557583 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.557515    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.557583 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.557529    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.557583 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.557543    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.557583 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.557556    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.558949 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.558923    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:06.559064 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.559019    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:56:06.559116 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.559103    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:06.559351 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.559323    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-kube-api-access-mtg6b" (OuterVolumeSpecName: "kube-api-access-mtg6b") pod "150c9d02-eb92-4e86-a874-d45b4cddb7b2" (UID: "150c9d02-eb92-4e86-a874-d45b4cddb7b2"). InnerVolumeSpecName "kube-api-access-mtg6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:06.657978 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.657934    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.657978 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.657972    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.657978 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.657983    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtg6b\" (UniqueName: \"kubernetes.io/projected/150c9d02-eb92-4e86-a874-d45b4cddb7b2-kube-api-access-mtg6b\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.657978 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.657991    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c9d02-eb92-4e86-a874-d45b4cddb7b2-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:06.678846 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.678816    2561 generic.go:356] "Generic (PLEG): container finished" podID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerID="e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7" exitCode=0
Sep 08 13:56:06.678946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.678860    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"
Sep 08 13:56:06.678946 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.678903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" event={"ID":"150c9d02-eb92-4e86-a874-d45b4cddb7b2","Type":"ContainerDied","Data":"e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7"}
Sep 08 13:56:06.679023 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.678951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j" event={"ID":"150c9d02-eb92-4e86-a874-d45b4cddb7b2","Type":"ContainerDied","Data":"7922490843d04cbcf8b473c024ad1a705f0780d3f7bb15280a087c106a2b3b12"}
Sep 08 13:56:06.679023 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.678974    2561 scope.go:117] "RemoveContainer" containerID="e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7"
Sep 08 13:56:06.688217 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.688197    2561 scope.go:117] "RemoveContainer" containerID="b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793"
Sep 08 13:56:06.695710 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.695690    2561 scope.go:117] "RemoveContainer" containerID="e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7"
Sep 08 13:56:06.695975 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:56:06.695955    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7\": container with ID starting with e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7 not found: ID does not exist" containerID="e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7"
Sep 08 13:56:06.696025 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.695983    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7"} err="failed to get container status \"e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7\": rpc error: code = NotFound desc = could not find container \"e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7\": container with ID starting with e35f61b11f66cf560362ec5e9b84606f1c82bb7a7d18e12740c7dc0ec0570ec7 not found: ID does not exist"
Sep 08 13:56:06.696025 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.696001    2561 scope.go:117] "RemoveContainer" containerID="b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793"
Sep 08 13:56:06.696221 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:56:06.696206    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793\": container with ID starting with b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793 not found: ID does not exist" containerID="b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793"
Sep 08 13:56:06.696259 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.696223    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793"} err="failed to get container status \"b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793\": rpc error: code = NotFound desc = could not find container \"b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793\": container with ID starting with b10a32d3e1a878ca2167aff847bcade013d131d778bafa9da9e4b9dfb6e96793 not found: ID does not exist"
Sep 08 13:56:06.700958 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.700935    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"]
Sep 08 13:56:06.706882 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:06.706860    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8ef041-openshift-default-689b55855d-s5n9j"]
Sep 08 13:56:08.245589 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:08.245556    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" path="/var/lib/kubelet/pods/150c9d02-eb92-4e86-a874-d45b4cddb7b2/volumes"
Sep 08 13:56:12.235091 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.235061    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"]
Sep 08 13:56:12.235513 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.235269    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr" podUID="4cabf638-c92a-4e30-b850-04babd96452c" containerName="payload-pre-processing" containerID="cri-o://efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434" gracePeriod=30
Sep 08 13:56:12.494314 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.494293    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:56:12.613624 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.613588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cabf638-c92a-4e30-b850-04babd96452c-plugins-config-volume\") pod \"4cabf638-c92a-4e30-b850-04babd96452c\" (UID: \"4cabf638-c92a-4e30-b850-04babd96452c\") "
Sep 08 13:56:12.613832 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.613659    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7j2r\" (UniqueName: \"kubernetes.io/projected/4cabf638-c92a-4e30-b850-04babd96452c-kube-api-access-v7j2r\") pod \"4cabf638-c92a-4e30-b850-04babd96452c\" (UID: \"4cabf638-c92a-4e30-b850-04babd96452c\") "
Sep 08 13:56:12.614036 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.614008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cabf638-c92a-4e30-b850-04babd96452c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cabf638-c92a-4e30-b850-04babd96452c" (UID: "4cabf638-c92a-4e30-b850-04babd96452c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:56:12.615742 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.615712    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cabf638-c92a-4e30-b850-04babd96452c-kube-api-access-v7j2r" (OuterVolumeSpecName: "kube-api-access-v7j2r") pod "4cabf638-c92a-4e30-b850-04babd96452c" (UID: "4cabf638-c92a-4e30-b850-04babd96452c"). InnerVolumeSpecName "kube-api-access-v7j2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:12.708914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.708874    2561 generic.go:356] "Generic (PLEG): container finished" podID="4cabf638-c92a-4e30-b850-04babd96452c" containerID="efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434" exitCode=0
Sep 08 13:56:12.708914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.708919    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"
Sep 08 13:56:12.709123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.708958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr" event={"ID":"4cabf638-c92a-4e30-b850-04babd96452c","Type":"ContainerDied","Data":"efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434"}
Sep 08 13:56:12.709123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.709011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr" event={"ID":"4cabf638-c92a-4e30-b850-04babd96452c","Type":"ContainerDied","Data":"b45268d3652297a5520fcf02ef76aec869752b090f00bba89e09f18b03757d0c"}
Sep 08 13:56:12.709123 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.709036    2561 scope.go:117] "RemoveContainer" containerID="efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434"
Sep 08 13:56:12.715339 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.715305    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cabf638-c92a-4e30-b850-04babd96452c-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:12.715483 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.715341    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7j2r\" (UniqueName: \"kubernetes.io/projected/4cabf638-c92a-4e30-b850-04babd96452c-kube-api-access-v7j2r\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:12.723109 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.723087    2561 scope.go:117] "RemoveContainer" containerID="efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434"
Sep 08 13:56:12.723457 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:56:12.723434    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434\": container with ID starting with efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434 not found: ID does not exist" containerID="efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434"
Sep 08 13:56:12.723506 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.723468    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434"} err="failed to get container status \"efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434\": rpc error: code = NotFound desc = could not find container \"efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434\": container with ID starting with efbaecbb1bc2b95cd400ed833e4d9d223e5297c411d833009a298ad17ebdf434 not found: ID does not exist"
Sep 08 13:56:12.733962 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.733939    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"]
Sep 08 13:56:12.738804 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:12.738751    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3c9fc8-66cfdc5f5d-cw8qr"]
Sep 08 13:56:14.244923 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:14.244887    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cabf638-c92a-4e30-b850-04babd96452c" path="/var/lib/kubelet/pods/4cabf638-c92a-4e30-b850-04babd96452c/volumes"
Sep 08 13:56:14.524113 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:14.524014    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"]
Sep 08 13:56:29.543782 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.543718    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h" podUID="937d2191-1010-4ae7-8454-84579de1a0be" containerName="main" containerID="cri-o://9ddfdd2d583f134614e975b0429802b36bd9c1063413f9b67627177778bf40bf" gracePeriod=45
Sep 08 13:56:29.789184 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.789145    2561 generic.go:356] "Generic (PLEG): container finished" podID="937d2191-1010-4ae7-8454-84579de1a0be" containerID="9ddfdd2d583f134614e975b0429802b36bd9c1063413f9b67627177778bf40bf" exitCode=0
Sep 08 13:56:29.789319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.789176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h" event={"ID":"937d2191-1010-4ae7-8454-84579de1a0be","Type":"ContainerDied","Data":"9ddfdd2d583f134614e975b0429802b36bd9c1063413f9b67627177778bf40bf"}
Sep 08 13:56:29.789319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.789220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h" event={"ID":"937d2191-1010-4ae7-8454-84579de1a0be","Type":"ContainerDied","Data":"f0af2923067bfa77d4e9a29258b73862f68219b25071a4c4ef95996f4a72a9eb"}
Sep 08 13:56:29.789319 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.789232    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0af2923067bfa77d4e9a29258b73862f68219b25071a4c4ef95996f4a72a9eb"
Sep 08 13:56:29.794121 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.794071    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:56:29.860303 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860273    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937d2191-1010-4ae7-8454-84579de1a0be-tls-certs\") pod \"937d2191-1010-4ae7-8454-84579de1a0be\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") "
Sep 08 13:56:29.860471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860318    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-home\") pod \"937d2191-1010-4ae7-8454-84579de1a0be\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") "
Sep 08 13:56:29.860471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-model-cache\") pod \"937d2191-1010-4ae7-8454-84579de1a0be\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") "
Sep 08 13:56:29.860471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860402    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqbrw\" (UniqueName: \"kubernetes.io/projected/937d2191-1010-4ae7-8454-84579de1a0be-kube-api-access-sqbrw\") pod \"937d2191-1010-4ae7-8454-84579de1a0be\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") "
Sep 08 13:56:29.860471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-dshm\") pod \"937d2191-1010-4ae7-8454-84579de1a0be\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") "
Sep 08 13:56:29.860471 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-tmp-dir\") pod \"937d2191-1010-4ae7-8454-84579de1a0be\" (UID: \"937d2191-1010-4ae7-8454-84579de1a0be\") "
Sep 08 13:56:29.860737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860620    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-home" (OuterVolumeSpecName: "home") pod "937d2191-1010-4ae7-8454-84579de1a0be" (UID: "937d2191-1010-4ae7-8454-84579de1a0be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:29.860737 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860635    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-model-cache" (OuterVolumeSpecName: "model-cache") pod "937d2191-1010-4ae7-8454-84579de1a0be" (UID: "937d2191-1010-4ae7-8454-84579de1a0be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:29.860857 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860736    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:29.860857 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860754    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:29.860857 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.860764    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "937d2191-1010-4ae7-8454-84579de1a0be" (UID: "937d2191-1010-4ae7-8454-84579de1a0be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:29.862445 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.862424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937d2191-1010-4ae7-8454-84579de1a0be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "937d2191-1010-4ae7-8454-84579de1a0be" (UID: "937d2191-1010-4ae7-8454-84579de1a0be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:56:29.862525 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.862476    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-dshm" (OuterVolumeSpecName: "dshm") pod "937d2191-1010-4ae7-8454-84579de1a0be" (UID: "937d2191-1010-4ae7-8454-84579de1a0be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:29.862620 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.862597    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937d2191-1010-4ae7-8454-84579de1a0be-kube-api-access-sqbrw" (OuterVolumeSpecName: "kube-api-access-sqbrw") pod "937d2191-1010-4ae7-8454-84579de1a0be" (UID: "937d2191-1010-4ae7-8454-84579de1a0be"). InnerVolumeSpecName "kube-api-access-sqbrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:29.961850 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.961801    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbrw\" (UniqueName: \"kubernetes.io/projected/937d2191-1010-4ae7-8454-84579de1a0be-kube-api-access-sqbrw\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:29.961850 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.961844    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:29.961850 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.961854    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937d2191-1010-4ae7-8454-84579de1a0be-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:29.961850 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:29.961863    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937d2191-1010-4ae7-8454-84579de1a0be-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:30.525955 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.525857    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"]
Sep 08 13:56:30.526174 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.526123    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r" podUID="86711db8-c910-4192-b453-fd6b6add55b7" containerName="payload-processing" containerID="cri-o://bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55" gracePeriod=30
Sep 08 13:56:30.775775 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.775751    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:56:30.796513 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.796367    2561 generic.go:356] "Generic (PLEG): container finished" podID="86711db8-c910-4192-b453-fd6b6add55b7" containerID="bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55" exitCode=0
Sep 08 13:56:30.796513 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.796445    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"
Sep 08 13:56:30.796513 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.796489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r" event={"ID":"86711db8-c910-4192-b453-fd6b6add55b7","Type":"ContainerDied","Data":"bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55"}
Sep 08 13:56:30.796823 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.796539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r" event={"ID":"86711db8-c910-4192-b453-fd6b6add55b7","Type":"ContainerDied","Data":"3005c185cdbe11a84e638fd11ab1597389480626d2dcb8355f27e01f8d1e8878"}
Sep 08 13:56:30.796823 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.796570    2561 scope.go:117] "RemoveContainer" containerID="bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55"
Sep 08 13:56:30.796823 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.796599    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"
Sep 08 13:56:30.805852 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.805830    2561 scope.go:117] "RemoveContainer" containerID="bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55"
Sep 08 13:56:30.806192 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:56:30.806172    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55\": container with ID starting with bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55 not found: ID does not exist" containerID="bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55"
Sep 08 13:56:30.806238 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.806201    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55"} err="failed to get container status \"bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55\": rpc error: code = NotFound desc = could not find container \"bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55\": container with ID starting with bcbd5b6b8113d4a6c836c91c310da197732bec82b161e9fb33a54e08169a5e55 not found: ID does not exist"
Sep 08 13:56:30.818107 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.818079    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"]
Sep 08 13:56:30.821743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.821720    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1f082e/rate-test-model-1f082e-kserve-5675d44c5f-gfb6h"]
Sep 08 13:56:30.869246 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.869220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86711db8-c910-4192-b453-fd6b6add55b7-plugins-config-volume\") pod \"86711db8-c910-4192-b453-fd6b6add55b7\" (UID: \"86711db8-c910-4192-b453-fd6b6add55b7\") "
Sep 08 13:56:30.869443 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.869323    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f4qz\" (UniqueName: \"kubernetes.io/projected/86711db8-c910-4192-b453-fd6b6add55b7-kube-api-access-5f4qz\") pod \"86711db8-c910-4192-b453-fd6b6add55b7\" (UID: \"86711db8-c910-4192-b453-fd6b6add55b7\") "
Sep 08 13:56:30.869767 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.869736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86711db8-c910-4192-b453-fd6b6add55b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86711db8-c910-4192-b453-fd6b6add55b7" (UID: "86711db8-c910-4192-b453-fd6b6add55b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:56:30.872150 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.872126    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86711db8-c910-4192-b453-fd6b6add55b7-kube-api-access-5f4qz" (OuterVolumeSpecName: "kube-api-access-5f4qz") pod "86711db8-c910-4192-b453-fd6b6add55b7" (UID: "86711db8-c910-4192-b453-fd6b6add55b7"). InnerVolumeSpecName "kube-api-access-5f4qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:30.970771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.970734    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4qz\" (UniqueName: \"kubernetes.io/projected/86711db8-c910-4192-b453-fd6b6add55b7-kube-api-access-5f4qz\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:30.970771 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:30.970764    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86711db8-c910-4192-b453-fd6b6add55b7-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:31.119697 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:31.119667    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"]
Sep 08 13:56:31.125000 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:31.124975    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9970e-6b67c8f675-lct9r"]
Sep 08 13:56:32.246043 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:32.246005    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86711db8-c910-4192-b453-fd6b6add55b7" path="/var/lib/kubelet/pods/86711db8-c910-4192-b453-fd6b6add55b7/volumes"
Sep 08 13:56:32.246468 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:32.246453    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="937d2191-1010-4ae7-8454-84579de1a0be" path="/var/lib/kubelet/pods/937d2191-1010-4ae7-8454-84579de1a0be/volumes"
Sep 08 13:56:37.317783 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:37.317697    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"]
Sep 08 13:56:37.318248 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:37.317964    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy" containerID="cri-o://f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb" gracePeriod=30
Sep 08 13:56:42.575584 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.575557    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:56:42.673041 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673004    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd7dc\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-kube-api-access-zd7dc\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673214 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-podinfo\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673214 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673115    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-credential-socket\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673214 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673143    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-envoy\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673214 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-socket\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673553 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673237    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-data\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673553 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-certs\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673553 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673297    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1575e4ae-d4c0-4007-adcc-29b52a10d532-istiod-ca-cert\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673553 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-token\") pod \"1575e4ae-d4c0-4007-adcc-29b52a10d532\" (UID: \"1575e4ae-d4c0-4007-adcc-29b52a10d532\") "
Sep 08 13:56:42.673553 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673474    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:42.673818 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:42.673818 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-data" (OuterVolumeSpecName: "istio-data") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:42.673818 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673779    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.673818 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673800    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.673818 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673815    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.674141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.673932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:42.674141 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.674111    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1575e4ae-d4c0-4007-adcc-29b52a10d532-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:56:42.675579 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.675548    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:42.675579 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.675560    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:56:42.675719 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.675595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-kube-api-access-zd7dc" (OuterVolumeSpecName: "kube-api-access-zd7dc") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "kube-api-access-zd7dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:42.675719 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.675653    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-token" (OuterVolumeSpecName: "istio-token") pod "1575e4ae-d4c0-4007-adcc-29b52a10d532" (UID: "1575e4ae-d4c0-4007-adcc-29b52a10d532"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:42.774992 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.774962    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.774992 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.774990    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.774992 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.774999    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1575e4ae-d4c0-4007-adcc-29b52a10d532-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.775230 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.775007    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1575e4ae-d4c0-4007-adcc-29b52a10d532-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.775230 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.775015    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.775230 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.775023    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7dc\" (UniqueName: \"kubernetes.io/projected/1575e4ae-d4c0-4007-adcc-29b52a10d532-kube-api-access-zd7dc\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:56:42.854639 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.854604    2561 generic.go:356] "Generic (PLEG): container finished" podID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerID="f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb" exitCode=0
Sep 08 13:56:42.854813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.854648    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" event={"ID":"1575e4ae-d4c0-4007-adcc-29b52a10d532","Type":"ContainerDied","Data":"f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb"}
Sep 08 13:56:42.854813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.854659    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"
Sep 08 13:56:42.854813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.854691    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j" event={"ID":"1575e4ae-d4c0-4007-adcc-29b52a10d532","Type":"ContainerDied","Data":"dd964e86d80c7513a8c5b6961f23d7d4ff11f8b68ee46513e9c5562caeee8660"}
Sep 08 13:56:42.854813 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.854715    2561 scope.go:117] "RemoveContainer" containerID="f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb"
Sep 08 13:56:42.864167 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.864148    2561 scope.go:117] "RemoveContainer" containerID="f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927"
Sep 08 13:56:42.872105 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.872088    2561 scope.go:117] "RemoveContainer" containerID="f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb"
Sep 08 13:56:42.872353 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:56:42.872338    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb\": container with ID starting with f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb not found: ID does not exist" containerID="f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb"
Sep 08 13:56:42.872434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.872363    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb"} err="failed to get container status \"f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb\": rpc error: code = NotFound desc = could not find container \"f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb\": container with ID starting with f7d855dd6dcbb309d6520888c2bac2eef2a30482c33ec121ebd409f5cdf4cffb not found: ID does not exist"
Sep 08 13:56:42.872434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.872394    2561 scope.go:117] "RemoveContainer" containerID="f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927"
Sep 08 13:56:42.872642 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:56:42.872618    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927\": container with ID starting with f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927 not found: ID does not exist" containerID="f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927"
Sep 08 13:56:42.872704 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.872651    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927"} err="failed to get container status \"f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927\": rpc error: code = NotFound desc = could not find container \"f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927\": container with ID starting with f86b8d99e6a66c152c677256b54932f10148ba4a0894b8ef6fbbf8e272c28927 not found: ID does not exist"
Sep 08 13:56:42.876155 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.876135    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"]
Sep 08 13:56:42.881743 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:42.881721    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d9970e-openshift-default-7ffc847f44-t566j"]
Sep 08 13:56:44.245022 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:44.244987    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" path="/var/lib/kubelet/pods/1575e4ae-d4c0-4007-adcc-29b52a10d532/volumes"
Sep 08 13:56:45.582560 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582528    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"]
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582833    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5e5348-253a-41d0-9907-d6fdae3ea4f6" containerName="revoke-keys"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582844    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5e5348-253a-41d0-9907-d6fdae3ea4f6" containerName="revoke-keys"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582862    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582868    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582874    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582879    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582886    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cabf638-c92a-4e30-b850-04babd96452c" containerName="payload-pre-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582892    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cabf638-c92a-4e30-b850-04babd96452c" containerName="payload-pre-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582897    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86711db8-c910-4192-b453-fd6b6add55b7" containerName="payload-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582902    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="86711db8-c910-4192-b453-fd6b6add55b7" containerName="payload-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582915    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937d2191-1010-4ae7-8454-84579de1a0be" containerName="main"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582920    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="937d2191-1010-4ae7-8454-84579de1a0be" containerName="main"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582929    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f5ad4d-b900-4e98-b7ee-1067883d1ada" containerName="payload-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582934    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f5ad4d-b900-4e98-b7ee-1067883d1ada" containerName="payload-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582980    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582986    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.582994    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cabf638-c92a-4e30-b850-04babd96452c" containerName="payload-pre-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.583000    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.583006    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="937d2191-1010-4ae7-8454-84579de1a0be" containerName="main"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.583014    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f5ad4d-b900-4e98-b7ee-1067883d1ada" containerName="payload-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.583019    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="86711db8-c910-4192-b453-fd6b6add55b7" containerName="payload-processing"
Sep 08 13:56:45.583037 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.583024    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5e5348-253a-41d0-9907-d6fdae3ea4f6" containerName="revoke-keys"
Sep 08 13:56:45.585824 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.585805    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.588521 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.588501    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cdf4f2-openshift-default-dockercfg-2ds6f\""
Sep 08 13:56:45.597872 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.597847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"]
Sep 08 13:56:45.700677 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.700639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-envoy\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.700914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.700694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-socket\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.700914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.700739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-certs\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.700914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.700766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlxhl\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-kube-api-access-rlxhl\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.700914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.700839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-istio-token\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.700914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.700900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-data\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.701160 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.700925    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe68d607-a87a-4aad-a976-94439d5d793f-istiod-ca-cert\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.701160 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.700970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe68d607-a87a-4aad-a976-94439d5d793f-istio-podinfo\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.701160 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.701082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-credential-socket\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.801984 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.801946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-istio-token\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.801984 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.801989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-data\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802252 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe68d607-a87a-4aad-a976-94439d5d793f-istiod-ca-cert\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802252 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe68d607-a87a-4aad-a976-94439d5d793f-istio-podinfo\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802252 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-credential-socket\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802252 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-envoy\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802252 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-socket\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802252 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-certs\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802252 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlxhl\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-kube-api-access-rlxhl\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802665 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-data\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802665 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-credential-socket\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802665 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-socket\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802776 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802695    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-certs\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.802903 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.802881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe68d607-a87a-4aad-a976-94439d5d793f-istiod-ca-cert\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.804421 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.804399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-envoy\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.804663 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.804647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe68d607-a87a-4aad-a976-94439d5d793f-istio-podinfo\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.810175 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.810148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlxhl\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-kube-api-access-rlxhl\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.810493 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.810478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-istio-token\") pod \"e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") " pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:45.895864 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:45.895779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:46.174916 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:46.174890    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"]
Sep 08 13:56:46.177460 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:56:46.177432    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe68d607_a87a_4aad_a976_94439d5d793f.slice/crio-d0075becfccb0215aaa1807dfd9be772bfa322b74d9c9a36b70f982db6732171 WatchSource:0}: Error finding container d0075becfccb0215aaa1807dfd9be772bfa322b74d9c9a36b70f982db6732171: Status 404 returned error can't find the container with id d0075becfccb0215aaa1807dfd9be772bfa322b74d9c9a36b70f982db6732171
Sep 08 13:56:46.179748 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:46.179716    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:56:46.179832 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:46.179780    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:56:46.179832 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:46.179806    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:56:46.876800 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:46.876756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" event={"ID":"fe68d607-a87a-4aad-a976-94439d5d793f","Type":"ContainerStarted","Data":"ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af"}
Sep 08 13:56:46.877292 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:46.876820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" event={"ID":"fe68d607-a87a-4aad-a976-94439d5d793f","Type":"ContainerStarted","Data":"d0075becfccb0215aaa1807dfd9be772bfa322b74d9c9a36b70f982db6732171"}
Sep 08 13:56:46.896521 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:46.896494    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:46.900062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:46.900018    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" podStartSLOduration=1.900003887 podStartE2EDuration="1.900003887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:56:46.898562385 +0000 UTC m=+1715.249548334" watchObservedRunningTime="2026-09-08 13:56:46.900003887 +0000 UTC m=+1715.250989906"
Sep 08 13:56:47.901504 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:47.901474    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:48.886740 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:48.886700    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:48.888213 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:48.888188    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:56:50.740098 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.740064    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"]
Sep 08 13:56:50.740490 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.740370    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy"
Sep 08 13:56:50.740490 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.740393    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy"
Sep 08 13:56:50.740490 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.740403    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy"
Sep 08 13:56:50.740490 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.740410    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1575e4ae-d4c0-4007-adcc-29b52a10d532" containerName="istio-proxy"
Sep 08 13:56:50.740490 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.740472    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="150c9d02-eb92-4e86-a874-d45b4cddb7b2" containerName="istio-proxy"
Sep 08 13:56:50.743477 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.743451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:50.746052 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.746033    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cdf4f2\""
Sep 08 13:56:50.746160 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.746068    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cdf4f2-dockercfg-nq5tl\""
Sep 08 13:56:50.754056 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.754033    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"]
Sep 08 13:56:50.844103 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.844070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gnx\" (UniqueName: \"kubernetes.io/projected/bb184b21-2640-42c2-9fba-ce427134e957-kube-api-access-n4gnx\") pod \"payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr\" (UID: \"bb184b21-2640-42c2-9fba-ce427134e957\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:50.844261 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.844115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb184b21-2640-42c2-9fba-ce427134e957-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr\" (UID: \"bb184b21-2640-42c2-9fba-ce427134e957\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:50.944613 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.944578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gnx\" (UniqueName: \"kubernetes.io/projected/bb184b21-2640-42c2-9fba-ce427134e957-kube-api-access-n4gnx\") pod \"payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr\" (UID: \"bb184b21-2640-42c2-9fba-ce427134e957\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:50.944613 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.944617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb184b21-2640-42c2-9fba-ce427134e957-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr\" (UID: \"bb184b21-2640-42c2-9fba-ce427134e957\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:50.945309 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.945290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb184b21-2640-42c2-9fba-ce427134e957-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr\" (UID: \"bb184b21-2640-42c2-9fba-ce427134e957\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:50.953603 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:50.953570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gnx\" (UniqueName: \"kubernetes.io/projected/bb184b21-2640-42c2-9fba-ce427134e957-kube-api-access-n4gnx\") pod \"payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr\" (UID: \"bb184b21-2640-42c2-9fba-ce427134e957\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:51.054753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:51.054677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:51.344640 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:51.344614    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"]
Sep 08 13:56:51.347261 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:56:51.347236    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb184b21_2640_42c2_9fba_ce427134e957.slice/crio-7f17168feeb7007709a0a97b2f8197a52bd85f26f3d0b644870c7fb297d40b1e WatchSource:0}: Error finding container 7f17168feeb7007709a0a97b2f8197a52bd85f26f3d0b644870c7fb297d40b1e: Status 404 returned error can't find the container with id 7f17168feeb7007709a0a97b2f8197a52bd85f26f3d0b644870c7fb297d40b1e
Sep 08 13:56:51.901871 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:51.901832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr" event={"ID":"bb184b21-2640-42c2-9fba-ce427134e957","Type":"ContainerStarted","Data":"20a5aabb79ff685ac06ec00ad39a174659d6ed68ffb02d1561be8dba713aa1ed"}
Sep 08 13:56:51.902298 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:51.901912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr" event={"ID":"bb184b21-2640-42c2-9fba-ce427134e957","Type":"ContainerStarted","Data":"7f17168feeb7007709a0a97b2f8197a52bd85f26f3d0b644870c7fb297d40b1e"}
Sep 08 13:56:51.902298 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:51.901982    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:56:51.920725 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:56:51.920677    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr" podStartSLOduration=1.9206627250000001 podStartE2EDuration="1.920662725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:56:51.91922817 +0000 UTC m=+1720.270214118" watchObservedRunningTime="2026-09-08 13:56:51.920662725 +0000 UTC m=+1720.271648673"
Sep 08 13:57:02.908954 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:02.908914    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 13:57:04.495693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:04.495025    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": read tcp 10.132.0.2:48838->10.132.0.79:15021: read: connection reset by peer" start-of-body=
Sep 08 13:57:04.495693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:04.495092    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": read tcp 10.132.0.2:48838->10.132.0.79:15021: read: connection reset by peer"
Sep 08 13:57:04.969979 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:04.969947    2561 generic.go:356] "Generic (PLEG): container finished" podID="fe68d607-a87a-4aad-a976-94439d5d793f" containerID="ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af" exitCode=0
Sep 08 13:57:04.970153 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:04.970020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" event={"ID":"fe68d607-a87a-4aad-a976-94439d5d793f","Type":"ContainerDied","Data":"ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af"}
Sep 08 13:57:04.970427 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:04.970412    2561 scope.go:117] "RemoveContainer" containerID="ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af"
Sep 08 13:57:04.971551 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:04.971526    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:57:04.971617 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:04.971589    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:57:04.971659 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:04.971620    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 08 13:57:05.976717 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:05.976672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" event={"ID":"fe68d607-a87a-4aad-a976-94439d5d793f","Type":"ContainerStarted","Data":"9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054"}
Sep 08 13:57:06.896166 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:06.896127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:57:06.896369 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:06.896345    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 08 13:57:06.896452 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:06.896429    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 08 13:57:07.896741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:07.896702    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 08 13:57:07.897131 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:07.896767    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 08 13:57:08.896474 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:08.896436    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 08 13:57:08.896666 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:08.896514    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 08 13:57:09.896984 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:09.896945    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 08 13:57:09.897411 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:09.897001    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 08 13:57:10.896600 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:10.896558    2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 08 13:57:10.896790 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:10.896638    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 08 13:57:11.899711 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:11.899678    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:57:11.900200 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:11.900059    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:57:11.900632 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:11.900613    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 13:57:19.348063 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:19.348031    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr_bb184b21-2640-42c2-9fba-ce427134e957/payload-processing/0.log"
Sep 08 13:57:29.139416 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.139365    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"]
Sep 08 13:57:29.142951 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.142933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.147265 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.147236    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2199d0\"/\"kube-root-ca.crt\""
Sep 08 13:57:29.147439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.147284    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2199d0\"/\"openshift-service-ca.crt\""
Sep 08 13:57:29.147439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.147307    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2199d0\"/\"ipp-route-2199d0-kserve-self-signed-certs\""
Sep 08 13:57:29.147439 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.147308    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2199d0\"/\"default-dockercfg-mnmnd\""
Sep 08 13:57:29.152820 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.152796    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"]
Sep 08 13:57:29.181664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.181630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-tmp-dir\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.181664 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.181667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-model-cache\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.181859 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.181750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-dshm\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.181859 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.181795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6556e746-6d55-4b15-abf5-279310525306-tls-certs\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.181859 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.181832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpq4r\" (UniqueName: \"kubernetes.io/projected/6556e746-6d55-4b15-abf5-279310525306-kube-api-access-vpq4r\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.181859 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.181849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-home\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.282918 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.282880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-dshm\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.282918 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.282923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6556e746-6d55-4b15-abf5-279310525306-tls-certs\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.283129 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.283060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpq4r\" (UniqueName: \"kubernetes.io/projected/6556e746-6d55-4b15-abf5-279310525306-kube-api-access-vpq4r\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.283129 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.283100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-home\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.283222 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.283145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-tmp-dir\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.283222 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.283174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-model-cache\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.283538 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.283517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-home\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.283633 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.283559    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-tmp-dir\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.283633 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.283576    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-model-cache\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.285652 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.285633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-dshm\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.286029 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.286009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6556e746-6d55-4b15-abf5-279310525306-tls-certs\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.293166 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.293145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpq4r\" (UniqueName: \"kubernetes.io/projected/6556e746-6d55-4b15-abf5-279310525306-kube-api-access-vpq4r\") pod \"ipp-route-2199d0-kserve-7dbbf87d7-f7r97\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") " pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.454173 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.454136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:29.724914 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:29.724893    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"]
Sep 08 13:57:29.727095 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:57:29.727067    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6556e746_6d55_4b15_abf5_279310525306.slice/crio-51f7e87cb78dd6b6bb2e9a55d669ed5ba8a82d54efbeb129ae290af409c3484d WatchSource:0}: Error finding container 51f7e87cb78dd6b6bb2e9a55d669ed5ba8a82d54efbeb129ae290af409c3484d: Status 404 returned error can't find the container with id 51f7e87cb78dd6b6bb2e9a55d669ed5ba8a82d54efbeb129ae290af409c3484d
Sep 08 13:57:30.093920 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:30.093832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97" event={"ID":"6556e746-6d55-4b15-abf5-279310525306","Type":"ContainerStarted","Data":"73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0"}
Sep 08 13:57:30.093920 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:30.093880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97" event={"ID":"6556e746-6d55-4b15-abf5-279310525306","Type":"ContainerStarted","Data":"51f7e87cb78dd6b6bb2e9a55d669ed5ba8a82d54efbeb129ae290af409c3484d"}
Sep 08 13:57:30.112339 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:30.112291    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97" podStartSLOduration=1.112277174 podStartE2EDuration="1.112277174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:57:30.111253095 +0000 UTC m=+1758.462239044" watchObservedRunningTime="2026-09-08 13:57:30.112277174 +0000 UTC m=+1758.463263121"
Sep 08 13:57:39.455284 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:39.455249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:39.455933 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:39.455308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:39.468520 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:39.468491    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:40.153644 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:40.153612    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:57:55.337323 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:57:55.337296    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr_bb184b21-2640-42c2-9fba-ce427134e957/payload-processing/0.log"
Sep 08 13:58:00.777031 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:00.776993    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"]
Sep 08 13:58:00.780434 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:00.780414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:00.784538 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:00.784516    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c780a1-dockercfg-xfn8x\""
Sep 08 13:58:00.785004 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:00.784984    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c780a1\""
Sep 08 13:58:00.793208 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:00.793185    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"]
Sep 08 13:58:00.964521 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:00.964477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb6rx\" (UniqueName: \"kubernetes.io/projected/211dffe4-4504-4f88-9ecb-0a69b231c86a-kube-api-access-cb6rx\") pod \"payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r\" (UID: \"211dffe4-4504-4f88-9ecb-0a69b231c86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:00.964712 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:00.964531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211dffe4-4504-4f88-9ecb-0a69b231c86a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r\" (UID: \"211dffe4-4504-4f88-9ecb-0a69b231c86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:01.065422 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:01.065313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb6rx\" (UniqueName: \"kubernetes.io/projected/211dffe4-4504-4f88-9ecb-0a69b231c86a-kube-api-access-cb6rx\") pod \"payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r\" (UID: \"211dffe4-4504-4f88-9ecb-0a69b231c86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:01.065422 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:01.065357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211dffe4-4504-4f88-9ecb-0a69b231c86a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r\" (UID: \"211dffe4-4504-4f88-9ecb-0a69b231c86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:01.066192 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:01.066168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211dffe4-4504-4f88-9ecb-0a69b231c86a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r\" (UID: \"211dffe4-4504-4f88-9ecb-0a69b231c86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:01.073582 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:01.073559    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb6rx\" (UniqueName: \"kubernetes.io/projected/211dffe4-4504-4f88-9ecb-0a69b231c86a-kube-api-access-cb6rx\") pod \"payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r\" (UID: \"211dffe4-4504-4f88-9ecb-0a69b231c86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:01.090315 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:01.090293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:01.360818 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:01.360794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"]
Sep 08 13:58:01.362989 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:58:01.362953    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211dffe4_4504_4f88_9ecb_0a69b231c86a.slice/crio-d774cfcf255373f6d9404bd6979a5d2d945a95d8a2364ce74619e966cd70b0a3 WatchSource:0}: Error finding container d774cfcf255373f6d9404bd6979a5d2d945a95d8a2364ce74619e966cd70b0a3: Status 404 returned error can't find the container with id d774cfcf255373f6d9404bd6979a5d2d945a95d8a2364ce74619e966cd70b0a3
Sep 08 13:58:02.251709 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:02.251669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r" event={"ID":"211dffe4-4504-4f88-9ecb-0a69b231c86a","Type":"ContainerStarted","Data":"5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6"}
Sep 08 13:58:02.252104 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:02.251729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r" event={"ID":"211dffe4-4504-4f88-9ecb-0a69b231c86a","Type":"ContainerStarted","Data":"d774cfcf255373f6d9404bd6979a5d2d945a95d8a2364ce74619e966cd70b0a3"}
Sep 08 13:58:02.252104 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:02.251798    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:02.271210 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:02.271154    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r" podStartSLOduration=2.271137915 podStartE2EDuration="2.271137915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:58:02.269655954 +0000 UTC m=+1790.620641901" watchObservedRunningTime="2026-09-08 13:58:02.271137915 +0000 UTC m=+1790.622123860"
Sep 08 13:58:13.258568 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:13.258532    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:30.726102 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:30.726065    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"]
Sep 08 13:58:30.726603 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:30.726318    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r" podUID="211dffe4-4504-4f88-9ecb-0a69b231c86a" containerName="payload-processing" containerID="cri-o://5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6" gracePeriod=30
Sep 08 13:58:31.000876 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.000843    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:31.105224 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.105192    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb6rx\" (UniqueName: \"kubernetes.io/projected/211dffe4-4504-4f88-9ecb-0a69b231c86a-kube-api-access-cb6rx\") pod \"211dffe4-4504-4f88-9ecb-0a69b231c86a\" (UID: \"211dffe4-4504-4f88-9ecb-0a69b231c86a\") "
Sep 08 13:58:31.105452 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.105313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211dffe4-4504-4f88-9ecb-0a69b231c86a-plugins-config-volume\") pod \"211dffe4-4504-4f88-9ecb-0a69b231c86a\" (UID: \"211dffe4-4504-4f88-9ecb-0a69b231c86a\") "
Sep 08 13:58:31.105713 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.105690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211dffe4-4504-4f88-9ecb-0a69b231c86a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "211dffe4-4504-4f88-9ecb-0a69b231c86a" (UID: "211dffe4-4504-4f88-9ecb-0a69b231c86a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:58:31.107261 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.107241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211dffe4-4504-4f88-9ecb-0a69b231c86a-kube-api-access-cb6rx" (OuterVolumeSpecName: "kube-api-access-cb6rx") pod "211dffe4-4504-4f88-9ecb-0a69b231c86a" (UID: "211dffe4-4504-4f88-9ecb-0a69b231c86a"). InnerVolumeSpecName "kube-api-access-cb6rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:58:31.206170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.206141    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb6rx\" (UniqueName: \"kubernetes.io/projected/211dffe4-4504-4f88-9ecb-0a69b231c86a-kube-api-access-cb6rx\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:58:31.206170 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.206166    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211dffe4-4504-4f88-9ecb-0a69b231c86a-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:58:31.385753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.385670    2561 generic.go:356] "Generic (PLEG): container finished" podID="211dffe4-4504-4f88-9ecb-0a69b231c86a" containerID="5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6" exitCode=0
Sep 08 13:58:31.385753 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.385710    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"
Sep 08 13:58:31.385941 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.385751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r" event={"ID":"211dffe4-4504-4f88-9ecb-0a69b231c86a","Type":"ContainerDied","Data":"5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6"}
Sep 08 13:58:31.385941 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.385804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r" event={"ID":"211dffe4-4504-4f88-9ecb-0a69b231c86a","Type":"ContainerDied","Data":"d774cfcf255373f6d9404bd6979a5d2d945a95d8a2364ce74619e966cd70b0a3"}
Sep 08 13:58:31.385941 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.385827    2561 scope.go:117] "RemoveContainer" containerID="5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6"
Sep 08 13:58:31.395233 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.395213    2561 scope.go:117] "RemoveContainer" containerID="5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6"
Sep 08 13:58:31.395542 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:58:31.395523    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6\": container with ID starting with 5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6 not found: ID does not exist" containerID="5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6"
Sep 08 13:58:31.395611 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.395551    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6"} err="failed to get container status \"5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6\": rpc error: code = NotFound desc = could not find container \"5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6\": container with ID starting with 5e0b7bc30a657c4102d5c45c220472d3d1f21877bd780b2e3084a0274f8b2ac6 not found: ID does not exist"
Sep 08 13:58:31.407497 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.407471    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"]
Sep 08 13:58:31.411589 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:31.411566    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c780a1-8465c6cbff-hcr6r"]
Sep 08 13:58:32.245653 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:32.245610    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="211dffe4-4504-4f88-9ecb-0a69b231c86a" path="/var/lib/kubelet/pods/211dffe4-4504-4f88-9ecb-0a69b231c86a/volumes"
Sep 08 13:58:41.082879 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.082836    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"]
Sep 08 13:58:41.083442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.083326    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211dffe4-4504-4f88-9ecb-0a69b231c86a" containerName="payload-processing"
Sep 08 13:58:41.083442 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.083345    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="211dffe4-4504-4f88-9ecb-0a69b231c86a" containerName="payload-processing"
Sep 08 13:58:41.083579 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.083462    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="211dffe4-4504-4f88-9ecb-0a69b231c86a" containerName="payload-processing"
Sep 08 13:58:41.087719 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.087698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"
Sep 08 13:58:41.090355 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.090331    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 13:58:41.090498 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.090479    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 13:58:41.093593 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.093572    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"]
Sep 08 13:58:41.191521 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.191485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e156e6-c54f-48ab-acaf-63eda58cf1de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf\" (UID: \"a5e156e6-c54f-48ab-acaf-63eda58cf1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"
Sep 08 13:58:41.292254 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.292216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e156e6-c54f-48ab-acaf-63eda58cf1de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf\" (UID: \"a5e156e6-c54f-48ab-acaf-63eda58cf1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"
Sep 08 13:58:41.292892 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.292867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e156e6-c54f-48ab-acaf-63eda58cf1de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf\" (UID: \"a5e156e6-c54f-48ab-acaf-63eda58cf1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"
Sep 08 13:58:41.399866 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.399792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"
Sep 08 13:58:41.673124 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:41.673098    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"]
Sep 08 13:58:41.676174 ip-10-0-130-208 kubenswrapper[2561]: W0908 13:58:41.676145    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e156e6_c54f_48ab_acaf_63eda58cf1de.slice/crio-593c1db6481406f6de98254d31ae68cf0af87c58613df194d7f84875e702388d WatchSource:0}: Error finding container 593c1db6481406f6de98254d31ae68cf0af87c58613df194d7f84875e702388d: Status 404 returned error can't find the container with id 593c1db6481406f6de98254d31ae68cf0af87c58613df194d7f84875e702388d
Sep 08 13:58:42.438665 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:42.438629    2561 generic.go:356] "Generic (PLEG): container finished" podID="a5e156e6-c54f-48ab-acaf-63eda58cf1de" containerID="cc29151878e19225932b5904b4b2bd59aafccaca9d8ff3f4fa947f7337621532" exitCode=0
Sep 08 13:58:42.439071 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:42.438706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf" event={"ID":"a5e156e6-c54f-48ab-acaf-63eda58cf1de","Type":"ContainerDied","Data":"cc29151878e19225932b5904b4b2bd59aafccaca9d8ff3f4fa947f7337621532"}
Sep 08 13:58:42.439071 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:42.438754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf" event={"ID":"a5e156e6-c54f-48ab-acaf-63eda58cf1de","Type":"ContainerStarted","Data":"593c1db6481406f6de98254d31ae68cf0af87c58613df194d7f84875e702388d"}
Sep 08 13:58:43.572105 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:43.572076    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"
Sep 08 13:58:43.711741 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:43.711654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e156e6-c54f-48ab-acaf-63eda58cf1de-maas-api-service-ca\") pod \"a5e156e6-c54f-48ab-acaf-63eda58cf1de\" (UID: \"a5e156e6-c54f-48ab-acaf-63eda58cf1de\") "
Sep 08 13:58:43.712012 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:43.711988    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e156e6-c54f-48ab-acaf-63eda58cf1de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5e156e6-c54f-48ab-acaf-63eda58cf1de" (UID: "a5e156e6-c54f-48ab-acaf-63eda58cf1de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:58:43.812673 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:43.812644    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e156e6-c54f-48ab-acaf-63eda58cf1de-maas-api-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:58:44.450829 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:44.450799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf" event={"ID":"a5e156e6-c54f-48ab-acaf-63eda58cf1de","Type":"ContainerDied","Data":"593c1db6481406f6de98254d31ae68cf0af87c58613df194d7f84875e702388d"}
Sep 08 13:58:44.451093 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:44.450842    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"
Sep 08 13:58:44.451093 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:44.450847    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="593c1db6481406f6de98254d31ae68cf0af87c58613df194d7f84875e702388d"
Sep 08 13:58:56.825403 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:56.825352    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"]
Sep 08 13:58:56.827295 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:56.827268    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2199d0-b2zhf"]
Sep 08 13:58:58.244943 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:58:58.244900    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e156e6-c54f-48ab-acaf-63eda58cf1de" path="/var/lib/kubelet/pods/a5e156e6-c54f-48ab-acaf-63eda58cf1de/volumes"
Sep 08 13:59:02.185515 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:02.185480    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"]
Sep 08 13:59:17.204985 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.204920    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97" podUID="6556e746-6d55-4b15-abf5-279310525306" containerName="main" containerID="cri-o://73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0" gracePeriod=45
Sep 08 13:59:17.460605 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.460541    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:59:17.497959 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.497923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpq4r\" (UniqueName: \"kubernetes.io/projected/6556e746-6d55-4b15-abf5-279310525306-kube-api-access-vpq4r\") pod \"6556e746-6d55-4b15-abf5-279310525306\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") "
Sep 08 13:59:17.498204 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.497986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-home\") pod \"6556e746-6d55-4b15-abf5-279310525306\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") "
Sep 08 13:59:17.498204 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.498032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-tmp-dir\") pod \"6556e746-6d55-4b15-abf5-279310525306\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") "
Sep 08 13:59:17.498204 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.498088    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-model-cache\") pod \"6556e746-6d55-4b15-abf5-279310525306\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") "
Sep 08 13:59:17.498204 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.498140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6556e746-6d55-4b15-abf5-279310525306-tls-certs\") pod \"6556e746-6d55-4b15-abf5-279310525306\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") "
Sep 08 13:59:17.498204 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.498178    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-dshm\") pod \"6556e746-6d55-4b15-abf5-279310525306\" (UID: \"6556e746-6d55-4b15-abf5-279310525306\") "
Sep 08 13:59:17.498679 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.498601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-model-cache" (OuterVolumeSpecName: "model-cache") pod "6556e746-6d55-4b15-abf5-279310525306" (UID: "6556e746-6d55-4b15-abf5-279310525306"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:17.498785 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.498724    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-home" (OuterVolumeSpecName: "home") pod "6556e746-6d55-4b15-abf5-279310525306" (UID: "6556e746-6d55-4b15-abf5-279310525306"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:17.498996 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.498844    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:59:17.498996 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.498869    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:59:17.499220 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.499183    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6556e746-6d55-4b15-abf5-279310525306" (UID: "6556e746-6d55-4b15-abf5-279310525306"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:17.500680 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.500641    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6556e746-6d55-4b15-abf5-279310525306-kube-api-access-vpq4r" (OuterVolumeSpecName: "kube-api-access-vpq4r") pod "6556e746-6d55-4b15-abf5-279310525306" (UID: "6556e746-6d55-4b15-abf5-279310525306"). InnerVolumeSpecName "kube-api-access-vpq4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:59:17.500986 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.500956    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6556e746-6d55-4b15-abf5-279310525306-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6556e746-6d55-4b15-abf5-279310525306" (UID: "6556e746-6d55-4b15-abf5-279310525306"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:59:17.501062 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.500971    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-dshm" (OuterVolumeSpecName: "dshm") pod "6556e746-6d55-4b15-abf5-279310525306" (UID: "6556e746-6d55-4b15-abf5-279310525306"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:17.599468 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.599428    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:59:17.599468 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.599463    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6556e746-6d55-4b15-abf5-279310525306-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:59:17.599468 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.599472    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6556e746-6d55-4b15-abf5-279310525306-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:59:17.599708 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.599484    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpq4r\" (UniqueName: \"kubernetes.io/projected/6556e746-6d55-4b15-abf5-279310525306-kube-api-access-vpq4r\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 13:59:17.608693 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.608663    2561 generic.go:356] "Generic (PLEG): container finished" podID="6556e746-6d55-4b15-abf5-279310525306" containerID="73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0" exitCode=0
Sep 08 13:59:17.608827 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.608707    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"
Sep 08 13:59:17.608827 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.608754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97" event={"ID":"6556e746-6d55-4b15-abf5-279310525306","Type":"ContainerDied","Data":"73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0"}
Sep 08 13:59:17.608827 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.608796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97" event={"ID":"6556e746-6d55-4b15-abf5-279310525306","Type":"ContainerDied","Data":"51f7e87cb78dd6b6bb2e9a55d669ed5ba8a82d54efbeb129ae290af409c3484d"}
Sep 08 13:59:17.608827 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.608813    2561 scope.go:117] "RemoveContainer" containerID="73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0"
Sep 08 13:59:17.617997 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.617978    2561 scope.go:117] "RemoveContainer" containerID="73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0"
Sep 08 13:59:17.618259 ip-10-0-130-208 kubenswrapper[2561]: E0908 13:59:17.618240    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0\": container with ID starting with 73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0 not found: ID does not exist" containerID="73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0"
Sep 08 13:59:17.618322 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.618270    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0"} err="failed to get container status \"73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0\": rpc error: code = NotFound desc = could not find container \"73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0\": container with ID starting with 73c75464f0adc7d7140faa219ece6300a4563cc0707b7ee3af4cade55af252a0 not found: ID does not exist"
Sep 08 13:59:17.630992 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.630966    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"]
Sep 08 13:59:17.635438 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:17.635415    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2199d0/ipp-route-2199d0-kserve-7dbbf87d7-f7r97"]
Sep 08 13:59:18.245869 ip-10-0-130-208 kubenswrapper[2561]: I0908 13:59:18.245826    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6556e746-6d55-4b15-abf5-279310525306" path="/var/lib/kubelet/pods/6556e746-6d55-4b15-abf5-279310525306/volumes"
Sep 08 14:00:00.144189 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.144149    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"]
Sep 08 14:00:00.144708 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.144640    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6556e746-6d55-4b15-abf5-279310525306" containerName="main"
Sep 08 14:00:00.144708 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.144659    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6556e746-6d55-4b15-abf5-279310525306" containerName="main"
Sep 08 14:00:00.144794 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.144714    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e156e6-c54f-48ab-acaf-63eda58cf1de" containerName="revoke-keys"
Sep 08 14:00:00.144794 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.144723    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e156e6-c54f-48ab-acaf-63eda58cf1de" containerName="revoke-keys"
Sep 08 14:00:00.144883 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.144796    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e156e6-c54f-48ab-acaf-63eda58cf1de" containerName="revoke-keys"
Sep 08 14:00:00.144883 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.144809    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6556e746-6d55-4b15-abf5-279310525306" containerName="main"
Sep 08 14:00:00.147930 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.147911    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"]
Sep 08 14:00:00.148052 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.148035    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"
Sep 08 14:00:00.151131 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.151111    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"
Sep 08 14:00:00.151277 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.151263    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 14:00:00.158829 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.158806    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"]
Sep 08 14:00:00.160179 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.160161    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"
Sep 08 14:00:00.160986 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.160964    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"]
Sep 08 14:00:00.169297 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.169275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"
Sep 08 14:00:00.440565 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.440536    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"]
Sep 08 14:00:00.442350 ip-10-0-130-208 kubenswrapper[2561]: W0908 14:00:00.442323    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657edf3c_6572_4ff4_9562_79c623632a87.slice/crio-3d5e1c7a199d94125d526845f81bd0b526cbe9a1fed775d9ef5e92c11831bddd WatchSource:0}: Error finding container 3d5e1c7a199d94125d526845f81bd0b526cbe9a1fed775d9ef5e92c11831bddd: Status 404 returned error can't find the container with id 3d5e1c7a199d94125d526845f81bd0b526cbe9a1fed775d9ef5e92c11831bddd
Sep 08 14:00:00.455711 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.455690    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"]
Sep 08 14:00:00.458164 ip-10-0-130-208 kubenswrapper[2561]: W0908 14:00:00.458140    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0cfb3f_aeec_430c_b4b8_6be887ea6349.slice/crio-74487e9c3b45183d4ecacb70236c3ee9121e97add0b248149943226ef46c95d3 WatchSource:0}: Error finding container 74487e9c3b45183d4ecacb70236c3ee9121e97add0b248149943226ef46c95d3: Status 404 returned error can't find the container with id 74487e9c3b45183d4ecacb70236c3ee9121e97add0b248149943226ef46c95d3
Sep 08 14:00:00.806440 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.806401    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh" event={"ID":"dc0cfb3f-aeec-430c-b4b8-6be887ea6349","Type":"ContainerStarted","Data":"d6e1467f3625e560e2ce5823b06fa5c6171f48dba7b8623ddf976d27a40fdc24"}
Sep 08 14:00:00.806620 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.806463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh" event={"ID":"dc0cfb3f-aeec-430c-b4b8-6be887ea6349","Type":"ContainerStarted","Data":"74487e9c3b45183d4ecacb70236c3ee9121e97add0b248149943226ef46c95d3"}
Sep 08 14:00:00.807918 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.807891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958" event={"ID":"657edf3c-6572-4ff4-9562-79c623632a87","Type":"ContainerStarted","Data":"95293d3dbde51838781929a2a86e1df16bfbd0efd753c4f0a6d37eaa60bf9929"}
Sep 08 14:00:00.808039 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.807939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958" event={"ID":"657edf3c-6572-4ff4-9562-79c623632a87","Type":"ContainerStarted","Data":"3d5e1c7a199d94125d526845f81bd0b526cbe9a1fed775d9ef5e92c11831bddd"}
Sep 08 14:00:00.823083 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.823034    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh" podStartSLOduration=0.823018679 podStartE2EDuration="823.018679ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:00:00.821522169 +0000 UTC m=+1909.172508115" watchObservedRunningTime="2026-09-08 14:00:00.823018679 +0000 UTC m=+1909.174004626"
Sep 08 14:00:00.837852 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:00.837800    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958" podStartSLOduration=0.83778117 podStartE2EDuration="837.78117ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:00:00.836078637 +0000 UTC m=+1909.187064584" watchObservedRunningTime="2026-09-08 14:00:00.83778117 +0000 UTC m=+1909.188767120"
Sep 08 14:00:01.814683 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:01.814647    2561 generic.go:356] "Generic (PLEG): container finished" podID="657edf3c-6572-4ff4-9562-79c623632a87" containerID="95293d3dbde51838781929a2a86e1df16bfbd0efd753c4f0a6d37eaa60bf9929" exitCode=0
Sep 08 14:00:01.815093 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:01.814726    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958" event={"ID":"657edf3c-6572-4ff4-9562-79c623632a87","Type":"ContainerDied","Data":"95293d3dbde51838781929a2a86e1df16bfbd0efd753c4f0a6d37eaa60bf9929"}
Sep 08 14:00:01.816172 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:01.816151    2561 generic.go:356] "Generic (PLEG): container finished" podID="dc0cfb3f-aeec-430c-b4b8-6be887ea6349" containerID="d6e1467f3625e560e2ce5823b06fa5c6171f48dba7b8623ddf976d27a40fdc24" exitCode=0
Sep 08 14:00:01.816288 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:01.816221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh" event={"ID":"dc0cfb3f-aeec-430c-b4b8-6be887ea6349","Type":"ContainerDied","Data":"d6e1467f3625e560e2ce5823b06fa5c6171f48dba7b8623ddf976d27a40fdc24"}
Sep 08 14:00:02.969896 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:02.969866    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"
Sep 08 14:00:02.992236 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:02.992209    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"
Sep 08 14:00:03.827972 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:03.827939    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"
Sep 08 14:00:03.827972 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:03.827961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958" event={"ID":"657edf3c-6572-4ff4-9562-79c623632a87","Type":"ContainerDied","Data":"3d5e1c7a199d94125d526845f81bd0b526cbe9a1fed775d9ef5e92c11831bddd"}
Sep 08 14:00:03.828196 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:03.828016    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d5e1c7a199d94125d526845f81bd0b526cbe9a1fed775d9ef5e92c11831bddd"
Sep 08 14:00:03.829136 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:03.829118    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"
Sep 08 14:00:03.829230 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:03.829135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh" event={"ID":"dc0cfb3f-aeec-430c-b4b8-6be887ea6349","Type":"ContainerDied","Data":"74487e9c3b45183d4ecacb70236c3ee9121e97add0b248149943226ef46c95d3"}
Sep 08 14:00:03.829230 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:03.829174    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74487e9c3b45183d4ecacb70236c3ee9121e97add0b248149943226ef46c95d3"
Sep 08 14:00:10.611841 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.611807    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"]
Sep 08 14:00:10.614681 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.614655    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cdf4f2-29814600-nvpmh"]
Sep 08 14:00:10.637334 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.637298    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"]
Sep 08 14:00:10.637573 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.637547    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr" podUID="bb184b21-2640-42c2-9fba-ce427134e957" containerName="payload-processing" containerID="cri-o://20a5aabb79ff685ac06ec00ad39a174659d6ed68ffb02d1561be8dba713aa1ed" gracePeriod=30
Sep 08 14:00:10.863190 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.863106    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb184b21-2640-42c2-9fba-ce427134e957" containerID="20a5aabb79ff685ac06ec00ad39a174659d6ed68ffb02d1561be8dba713aa1ed" exitCode=0
Sep 08 14:00:10.863320 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.863181    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr" event={"ID":"bb184b21-2640-42c2-9fba-ce427134e957","Type":"ContainerDied","Data":"20a5aabb79ff685ac06ec00ad39a174659d6ed68ffb02d1561be8dba713aa1ed"}
Sep 08 14:00:10.900521 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.900492    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 14:00:10.952680 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.952647    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb184b21-2640-42c2-9fba-ce427134e957-plugins-config-volume\") pod \"bb184b21-2640-42c2-9fba-ce427134e957\" (UID: \"bb184b21-2640-42c2-9fba-ce427134e957\") "
Sep 08 14:00:10.952838 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.952689    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4gnx\" (UniqueName: \"kubernetes.io/projected/bb184b21-2640-42c2-9fba-ce427134e957-kube-api-access-n4gnx\") pod \"bb184b21-2640-42c2-9fba-ce427134e957\" (UID: \"bb184b21-2640-42c2-9fba-ce427134e957\") "
Sep 08 14:00:10.953030 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.953006    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb184b21-2640-42c2-9fba-ce427134e957-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb184b21-2640-42c2-9fba-ce427134e957" (UID: "bb184b21-2640-42c2-9fba-ce427134e957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:10.954938 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:10.954918    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb184b21-2640-42c2-9fba-ce427134e957-kube-api-access-n4gnx" (OuterVolumeSpecName: "kube-api-access-n4gnx") pod "bb184b21-2640-42c2-9fba-ce427134e957" (UID: "bb184b21-2640-42c2-9fba-ce427134e957"). InnerVolumeSpecName "kube-api-access-n4gnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:11.053597 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:11.053561    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb184b21-2640-42c2-9fba-ce427134e957-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:11.053597 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:11.053594    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4gnx\" (UniqueName: \"kubernetes.io/projected/bb184b21-2640-42c2-9fba-ce427134e957-kube-api-access-n4gnx\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:11.869063 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:11.869029    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"
Sep 08 14:00:11.869569 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:11.869085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr" event={"ID":"bb184b21-2640-42c2-9fba-ce427134e957","Type":"ContainerDied","Data":"7f17168feeb7007709a0a97b2f8197a52bd85f26f3d0b644870c7fb297d40b1e"}
Sep 08 14:00:11.869569 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:11.869152    2561 scope.go:117] "RemoveContainer" containerID="20a5aabb79ff685ac06ec00ad39a174659d6ed68ffb02d1561be8dba713aa1ed"
Sep 08 14:00:11.891235 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:11.891206    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"]
Sep 08 14:00:11.895338 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:11.895313    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cdf4f2-677656f549-q9pbr"]
Sep 08 14:00:12.245125 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:12.245090    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb184b21-2640-42c2-9fba-ce427134e957" path="/var/lib/kubelet/pods/bb184b21-2640-42c2-9fba-ce427134e957/volumes"
Sep 08 14:00:12.245387 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:12.245364    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc0cfb3f-aeec-430c-b4b8-6be887ea6349" path="/var/lib/kubelet/pods/dc0cfb3f-aeec-430c-b4b8-6be887ea6349/volumes"
Sep 08 14:00:17.305326 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:17.305291    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"]
Sep 08 14:00:17.305807 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:17.305564    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy" containerID="cri-o://9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054" gracePeriod=30
Sep 08 14:00:22.546520 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.546489    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 14:00:22.654983 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.654908    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-credential-socket\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.654983 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.654944    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-istio-token\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.654983 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.654966    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-data\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.655263 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.654991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-certs\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.655263 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655118    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-socket\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.655263 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655180    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlxhl\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-kube-api-access-rlxhl\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.655263 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655201    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:22.655263 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655220    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:22.655263 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-envoy\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.655587 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655288    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe68d607-a87a-4aad-a976-94439d5d793f-istiod-ca-cert\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.655587 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe68d607-a87a-4aad-a976-94439d5d793f-istio-podinfo\") pod \"fe68d607-a87a-4aad-a976-94439d5d793f\" (UID: \"fe68d607-a87a-4aad-a976-94439d5d793f\") "
Sep 08 14:00:22.655587 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655335    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:22.655587 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655448    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:22.655792 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655608    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.655792 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655627    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.655792 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655642    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.655792 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655655    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.655792 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.655703    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe68d607-a87a-4aad-a976-94439d5d793f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:22.657109 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.657083    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:22.657487 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.657467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe68d607-a87a-4aad-a976-94439d5d793f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 14:00:22.657808 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.657787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-kube-api-access-rlxhl" (OuterVolumeSpecName: "kube-api-access-rlxhl") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "kube-api-access-rlxhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:22.657871 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.657847    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe68d607-a87a-4aad-a976-94439d5d793f" (UID: "fe68d607-a87a-4aad-a976-94439d5d793f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:22.756952 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.756920    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.756952 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.756946    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlxhl\" (UniqueName: \"kubernetes.io/projected/fe68d607-a87a-4aad-a976-94439d5d793f-kube-api-access-rlxhl\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.756952 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.756956    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe68d607-a87a-4aad-a976-94439d5d793f-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.757180 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.756964    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe68d607-a87a-4aad-a976-94439d5d793f-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.757180 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.756972    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe68d607-a87a-4aad-a976-94439d5d793f-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:22.922216 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.922181    2561 generic.go:356] "Generic (PLEG): container finished" podID="fe68d607-a87a-4aad-a976-94439d5d793f" containerID="9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054" exitCode=0
Sep 08 14:00:22.922411 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.922228    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"
Sep 08 14:00:22.922411 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.922268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" event={"ID":"fe68d607-a87a-4aad-a976-94439d5d793f","Type":"ContainerDied","Data":"9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054"}
Sep 08 14:00:22.922411 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.922323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6" event={"ID":"fe68d607-a87a-4aad-a976-94439d5d793f","Type":"ContainerDied","Data":"d0075becfccb0215aaa1807dfd9be772bfa322b74d9c9a36b70f982db6732171"}
Sep 08 14:00:22.922411 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.922346    2561 scope.go:117] "RemoveContainer" containerID="9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054"
Sep 08 14:00:22.931117 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.931096    2561 scope.go:117] "RemoveContainer" containerID="ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af"
Sep 08 14:00:22.938709 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.938693    2561 scope.go:117] "RemoveContainer" containerID="9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054"
Sep 08 14:00:22.938975 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:22.938957    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054\": container with ID starting with 9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054 not found: ID does not exist" containerID="9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054"
Sep 08 14:00:22.939018 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.938986    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054"} err="failed to get container status \"9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054\": rpc error: code = NotFound desc = could not find container \"9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054\": container with ID starting with 9dbf51b5858eabac7ac2e4b3ea0480edd093b462384785f5547e563a6cf65054 not found: ID does not exist"
Sep 08 14:00:22.939018 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.939002    2561 scope.go:117] "RemoveContainer" containerID="ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af"
Sep 08 14:00:22.939244 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:22.939227    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af\": container with ID starting with ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af not found: ID does not exist" containerID="ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af"
Sep 08 14:00:22.939284 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.939251    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af"} err="failed to get container status \"ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af\": rpc error: code = NotFound desc = could not find container \"ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af\": container with ID starting with ab7be088a746597b1756a4136cd2c8d060172b90172b86b4c4994e4e4c6c02af not found: ID does not exist"
Sep 08 14:00:22.944771 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.944751    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"]
Sep 08 14:00:22.949370 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:22.949347    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cdf4f2-openshift-default-59c448bf7d-stps6"]
Sep 08 14:00:24.244976 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:24.244943    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" path="/var/lib/kubelet/pods/fe68d607-a87a-4aad-a976-94439d5d793f/volumes"
Sep 08 14:00:28.407068 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.407026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"]
Sep 08 14:00:28.409299 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.409276    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1f082e-29814600-sd958"]
Sep 08 14:00:28.430610 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.430583    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"]
Sep 08 14:00:28.430818 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.430787    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88" podUID="f9aba5bf-0101-4121-a8dc-4933bffd79cb" containerName="payload-processing" containerID="cri-o://de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2" gracePeriod=30
Sep 08 14:00:28.439018 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.438990    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"]
Sep 08 14:00:28.439241 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.439215    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v" podUID="a2e15d50-e9f4-46a2-a8eb-c0e7378e8479" containerName="payload-pre-processing" containerID="cri-o://89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291" gracePeriod=30
Sep 08 14:00:28.717494 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.717471    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 14:00:28.721523 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.721500    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 14:00:28.806141 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.806109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9aba5bf-0101-4121-a8dc-4933bffd79cb-plugins-config-volume\") pod \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\" (UID: \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\") "
Sep 08 14:00:28.806317 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.806149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psvvd\" (UniqueName: \"kubernetes.io/projected/f9aba5bf-0101-4121-a8dc-4933bffd79cb-kube-api-access-psvvd\") pod \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\" (UID: \"f9aba5bf-0101-4121-a8dc-4933bffd79cb\") "
Sep 08 14:00:28.806317 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.806182    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-plugins-config-volume\") pod \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\" (UID: \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\") "
Sep 08 14:00:28.806468 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.806436    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4ddz\" (UniqueName: \"kubernetes.io/projected/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-kube-api-access-q4ddz\") pod \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\" (UID: \"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479\") "
Sep 08 14:00:28.806560 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.806540    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2e15d50-e9f4-46a2-a8eb-c0e7378e8479" (UID: "a2e15d50-e9f4-46a2-a8eb-c0e7378e8479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:28.806618 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.806585    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9aba5bf-0101-4121-a8dc-4933bffd79cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9aba5bf-0101-4121-a8dc-4933bffd79cb" (UID: "f9aba5bf-0101-4121-a8dc-4933bffd79cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:28.806781 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.806764    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:28.806841 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.806782    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9aba5bf-0101-4121-a8dc-4933bffd79cb-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:28.808232 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.808210    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9aba5bf-0101-4121-a8dc-4933bffd79cb-kube-api-access-psvvd" (OuterVolumeSpecName: "kube-api-access-psvvd") pod "f9aba5bf-0101-4121-a8dc-4933bffd79cb" (UID: "f9aba5bf-0101-4121-a8dc-4933bffd79cb"). InnerVolumeSpecName "kube-api-access-psvvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:28.808426 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.808407    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-kube-api-access-q4ddz" (OuterVolumeSpecName: "kube-api-access-q4ddz") pod "a2e15d50-e9f4-46a2-a8eb-c0e7378e8479" (UID: "a2e15d50-e9f4-46a2-a8eb-c0e7378e8479"). InnerVolumeSpecName "kube-api-access-q4ddz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:28.907733 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.907689    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4ddz\" (UniqueName: \"kubernetes.io/projected/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479-kube-api-access-q4ddz\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:28.907733 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.907724    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvvd\" (UniqueName: \"kubernetes.io/projected/f9aba5bf-0101-4121-a8dc-4933bffd79cb-kube-api-access-psvvd\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:28.950503 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.950413    2561 generic.go:356] "Generic (PLEG): container finished" podID="f9aba5bf-0101-4121-a8dc-4933bffd79cb" containerID="de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2" exitCode=0
Sep 08 14:00:28.950503 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.950477    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"
Sep 08 14:00:28.950503 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.950495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88" event={"ID":"f9aba5bf-0101-4121-a8dc-4933bffd79cb","Type":"ContainerDied","Data":"de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2"}
Sep 08 14:00:28.950761 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.950571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88" event={"ID":"f9aba5bf-0101-4121-a8dc-4933bffd79cb","Type":"ContainerDied","Data":"44a3b085300589adc5aca22bc46d09295cd58ab87baabf52e6f2c3c64ff2bce5"}
Sep 08 14:00:28.950761 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.950600    2561 scope.go:117] "RemoveContainer" containerID="de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2"
Sep 08 14:00:28.951842 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.951813    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2e15d50-e9f4-46a2-a8eb-c0e7378e8479" containerID="89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291" exitCode=0
Sep 08 14:00:28.951970 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.951877    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"
Sep 08 14:00:28.951970 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.951879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v" event={"ID":"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479","Type":"ContainerDied","Data":"89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291"}
Sep 08 14:00:28.952076 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.952000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v" event={"ID":"a2e15d50-e9f4-46a2-a8eb-c0e7378e8479","Type":"ContainerDied","Data":"64ac72a15748e5b88f7b7811028b47de53216210e9f440f8f8dbc65e62275eab"}
Sep 08 14:00:28.960074 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.960057    2561 scope.go:117] "RemoveContainer" containerID="de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2"
Sep 08 14:00:28.960360 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:28.960343    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2\": container with ID starting with de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2 not found: ID does not exist" containerID="de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2"
Sep 08 14:00:28.960506 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.960368    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2"} err="failed to get container status \"de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2\": rpc error: code = NotFound desc = could not find container \"de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2\": container with ID starting with de8991084841ef0bd3c1a905592df473dfdd863794d7f7e94c57ca63825a11e2 not found: ID does not exist"
Sep 08 14:00:28.960506 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.960413    2561 scope.go:117] "RemoveContainer" containerID="89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291"
Sep 08 14:00:28.967722 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.967704    2561 scope.go:117] "RemoveContainer" containerID="89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291"
Sep 08 14:00:28.967957 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:28.967938    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291\": container with ID starting with 89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291 not found: ID does not exist" containerID="89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291"
Sep 08 14:00:28.968002 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.967966    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291"} err="failed to get container status \"89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291\": rpc error: code = NotFound desc = could not find container \"89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291\": container with ID starting with 89e3acf5429c27b6e65cb0de815c9cfcf040c537a508748f6e1b4836f07d6291 not found: ID does not exist"
Sep 08 14:00:28.987739 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:28.987707    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"]
Sep 08 14:00:29.001577 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:29.001552    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1f082e-ff4956c8b-lk85v"]
Sep 08 14:00:29.025522 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:29.025490    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"]
Sep 08 14:00:29.037539 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:29.037509    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1f082e-76b5fbf7f9-spj88"]
Sep 08 14:00:30.245190 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:30.245152    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="657edf3c-6572-4ff4-9562-79c623632a87" path="/var/lib/kubelet/pods/657edf3c-6572-4ff4-9562-79c623632a87/volumes"
Sep 08 14:00:30.245801 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:30.245780    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e15d50-e9f4-46a2-a8eb-c0e7378e8479" path="/var/lib/kubelet/pods/a2e15d50-e9f4-46a2-a8eb-c0e7378e8479/volumes"
Sep 08 14:00:30.246170 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:30.246152    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9aba5bf-0101-4121-a8dc-4933bffd79cb" path="/var/lib/kubelet/pods/f9aba5bf-0101-4121-a8dc-4933bffd79cb/volumes"
Sep 08 14:00:35.175511 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.175477    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"]
Sep 08 14:00:35.176128 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.175741    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy" containerID="cri-o://b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec" gracePeriod=30
Sep 08 14:00:35.537194 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537107    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"]
Sep 08 14:00:35.537417 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537405    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9aba5bf-0101-4121-a8dc-4933bffd79cb" containerName="payload-processing"
Sep 08 14:00:35.537472 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537418    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9aba5bf-0101-4121-a8dc-4933bffd79cb" containerName="payload-processing"
Sep 08 14:00:35.537472 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537429    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0cfb3f-aeec-430c-b4b8-6be887ea6349" containerName="cleanup"
Sep 08 14:00:35.537472 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537434    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0cfb3f-aeec-430c-b4b8-6be887ea6349" containerName="cleanup"
Sep 08 14:00:35.537472 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537445    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy"
Sep 08 14:00:35.537472 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537450    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy"
Sep 08 14:00:35.537472 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537465    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657edf3c-6572-4ff4-9562-79c623632a87" containerName="cleanup"
Sep 08 14:00:35.537472 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537470    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="657edf3c-6572-4ff4-9562-79c623632a87" containerName="cleanup"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537478    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e15d50-e9f4-46a2-a8eb-c0e7378e8479" containerName="payload-pre-processing"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537484    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e15d50-e9f4-46a2-a8eb-c0e7378e8479" containerName="payload-pre-processing"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537495    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb184b21-2640-42c2-9fba-ce427134e957" containerName="payload-processing"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537501    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb184b21-2640-42c2-9fba-ce427134e957" containerName="payload-processing"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537547    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb184b21-2640-42c2-9fba-ce427134e957" containerName="payload-processing"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537554    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="657edf3c-6572-4ff4-9562-79c623632a87" containerName="cleanup"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537563    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9aba5bf-0101-4121-a8dc-4933bffd79cb" containerName="payload-processing"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537569    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537575    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e15d50-e9f4-46a2-a8eb-c0e7378e8479" containerName="payload-pre-processing"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537581    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0cfb3f-aeec-430c-b4b8-6be887ea6349" containerName="cleanup"
Sep 08 14:00:35.537687 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.537585    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy"
Sep 08 14:00:35.541936 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.541920    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"
Sep 08 14:00:35.544518 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.544495    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 14:00:35.544637 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.544612    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-44529\""
Sep 08 14:00:35.547661 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.547632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"]
Sep 08 14:00:35.664929 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.664889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5\" (UID: \"a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"
Sep 08 14:00:35.765578 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.765542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5\" (UID: \"a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"
Sep 08 14:00:35.766168 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.766146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5\" (UID: \"a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"
Sep 08 14:00:35.853534 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:35.853445    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"
Sep 08 14:00:36.124321 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:36.124297    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"]
Sep 08 14:00:36.126430 ip-10-0-130-208 kubenswrapper[2561]: W0908 14:00:36.126402    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda519ffe4_1ea5_4a5b_bcfb_4d9f141ce31c.slice/crio-654dcedf69ca432c5cd0b8495c54e34eec63775687f071e43f3ff321866aec70 WatchSource:0}: Error finding container 654dcedf69ca432c5cd0b8495c54e34eec63775687f071e43f3ff321866aec70: Status 404 returned error can't find the container with id 654dcedf69ca432c5cd0b8495c54e34eec63775687f071e43f3ff321866aec70
Sep 08 14:00:36.358708 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:36.358671    2561 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 14:00:36.359037 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:36.358737    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 14:00:36.989992 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:36.989953    2561 generic.go:356] "Generic (PLEG): container finished" podID="a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c" containerID="3b3e376926b7e01a65fa3d630265cdf8fab08d92d099fb72ee2d31c5f1a06f85" exitCode=0
Sep 08 14:00:36.990176 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:36.990035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5" event={"ID":"a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c","Type":"ContainerDied","Data":"3b3e376926b7e01a65fa3d630265cdf8fab08d92d099fb72ee2d31c5f1a06f85"}
Sep 08 14:00:36.990176 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:36.990102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5" event={"ID":"a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c","Type":"ContainerStarted","Data":"654dcedf69ca432c5cd0b8495c54e34eec63775687f071e43f3ff321866aec70"}
Sep 08 14:00:38.122679 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:38.122651    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"
Sep 08 14:00:38.287449 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:38.287341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c-maas-api-service-ca\") pod \"a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c\" (UID: \"a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c\") "
Sep 08 14:00:38.287741 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:38.287715    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c" (UID: "a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:38.388265 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:38.388230    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c-maas-api-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:39.000348 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:39.000321    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"
Sep 08 14:00:39.000539 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:39.000317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5" event={"ID":"a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c","Type":"ContainerDied","Data":"654dcedf69ca432c5cd0b8495c54e34eec63775687f071e43f3ff321866aec70"}
Sep 08 14:00:39.000539 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:39.000437    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="654dcedf69ca432c5cd0b8495c54e34eec63775687f071e43f3ff321866aec70"
Sep 08 14:00:40.424938 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.424914    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 14:00:40.507580 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-data\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507580 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-credential-socket\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507580 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507567    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-certs\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507860 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507608    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-podinfo\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507860 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-token\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507860 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507657    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-envoy\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507860 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507681    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl9j8\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-kube-api-access-gl9j8\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507860 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-socket\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507860 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50bf45d-2f90-4d53-bb87-6b8d98246502-istiod-ca-cert\") pod \"f50bf45d-2f90-4d53-bb87-6b8d98246502\" (UID: \"f50bf45d-2f90-4d53-bb87-6b8d98246502\") "
Sep 08 14:00:40.507860 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507834    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-data" (OuterVolumeSpecName: "istio-data") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:40.508193 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507850    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:40.508193 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.507865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:40.508193 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.508064    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:40.508193 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.508114    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.508193 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.508137    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.508193 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.508157    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.508444 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.508289    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50bf45d-2f90-4d53-bb87-6b8d98246502-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:40.510000 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.509976    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 14:00:40.510000 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.509984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:40.510261 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.510234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-token" (OuterVolumeSpecName: "istio-token") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:40.510362 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.510259    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-kube-api-access-gl9j8" (OuterVolumeSpecName: "kube-api-access-gl9j8") pod "f50bf45d-2f90-4d53-bb87-6b8d98246502" (UID: "f50bf45d-2f90-4d53-bb87-6b8d98246502"). InnerVolumeSpecName "kube-api-access-gl9j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:40.609360 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.609323    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.609360 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.609353    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl9j8\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-kube-api-access-gl9j8\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.609360 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.609364    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50bf45d-2f90-4d53-bb87-6b8d98246502-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.609622 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.609373    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50bf45d-2f90-4d53-bb87-6b8d98246502-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.609622 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.609418    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.609622 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:40.609426    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50bf45d-2f90-4d53-bb87-6b8d98246502-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:41.011889 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.011851    2561 generic.go:356] "Generic (PLEG): container finished" podID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerID="b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec" exitCode=0
Sep 08 14:00:41.012066 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.011902    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"
Sep 08 14:00:41.012066 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.011935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" event={"ID":"f50bf45d-2f90-4d53-bb87-6b8d98246502","Type":"ContainerDied","Data":"b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec"}
Sep 08 14:00:41.012066 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.011989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4" event={"ID":"f50bf45d-2f90-4d53-bb87-6b8d98246502","Type":"ContainerDied","Data":"bd47b30a3edeab2a24c0b39880df4494bb7f244aebef4d47d122699932ff6bf1"}
Sep 08 14:00:41.012066 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.012013    2561 scope.go:117] "RemoveContainer" containerID="b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec"
Sep 08 14:00:41.021088 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.021071    2561 scope.go:117] "RemoveContainer" containerID="2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9"
Sep 08 14:00:41.028667 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.028648    2561 scope.go:117] "RemoveContainer" containerID="b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec"
Sep 08 14:00:41.028958 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:41.028940    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec\": container with ID starting with b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec not found: ID does not exist" containerID="b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec"
Sep 08 14:00:41.029010 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.028967    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec"} err="failed to get container status \"b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec\": rpc error: code = NotFound desc = could not find container \"b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec\": container with ID starting with b50616c99160d31515f23b59614945d06df04599dad995e4b198bab44cf69eec not found: ID does not exist"
Sep 08 14:00:41.029010 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.028983    2561 scope.go:117] "RemoveContainer" containerID="2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9"
Sep 08 14:00:41.029224 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:41.029204    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9\": container with ID starting with 2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9 not found: ID does not exist" containerID="2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9"
Sep 08 14:00:41.029288 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.029231    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9"} err="failed to get container status \"2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9\": rpc error: code = NotFound desc = could not find container \"2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9\": container with ID starting with 2d41c1378c88152d0cdaf6f7930a9df44507c43aedeae59849d4a9afaf88aef9 not found: ID does not exist"
Sep 08 14:00:41.034259 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.034234    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"]
Sep 08 14:00:41.038539 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:41.038518    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1f082e-openshift-default-665868bc88-ghnp4"]
Sep 08 14:00:42.245192 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:42.245161    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" path="/var/lib/kubelet/pods/f50bf45d-2f90-4d53-bb87-6b8d98246502/volumes"
Sep 08 14:00:46.078861 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.078820    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"]
Sep 08 14:00:46.079399 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.079116    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w" podUID="7ce9cece-477a-4b17-92cd-dd8a1869d000" containerName="payload-pre-processing" containerID="cri-o://2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2" gracePeriod=30
Sep 08 14:00:46.337466 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.337404    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 14:00:46.355058 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.355029    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"]
Sep 08 14:00:46.363287 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.363257    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-905fec-vqpj5"]
Sep 08 14:00:46.457597 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.457558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfmr2\" (UniqueName: \"kubernetes.io/projected/7ce9cece-477a-4b17-92cd-dd8a1869d000-kube-api-access-qfmr2\") pod \"7ce9cece-477a-4b17-92cd-dd8a1869d000\" (UID: \"7ce9cece-477a-4b17-92cd-dd8a1869d000\") "
Sep 08 14:00:46.457772 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.457625    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce9cece-477a-4b17-92cd-dd8a1869d000-plugins-config-volume\") pod \"7ce9cece-477a-4b17-92cd-dd8a1869d000\" (UID: \"7ce9cece-477a-4b17-92cd-dd8a1869d000\") "
Sep 08 14:00:46.458022 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.457989    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce9cece-477a-4b17-92cd-dd8a1869d000-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ce9cece-477a-4b17-92cd-dd8a1869d000" (UID: "7ce9cece-477a-4b17-92cd-dd8a1869d000"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:46.459692 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.459669    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce9cece-477a-4b17-92cd-dd8a1869d000-kube-api-access-qfmr2" (OuterVolumeSpecName: "kube-api-access-qfmr2") pod "7ce9cece-477a-4b17-92cd-dd8a1869d000" (UID: "7ce9cece-477a-4b17-92cd-dd8a1869d000"). InnerVolumeSpecName "kube-api-access-qfmr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:46.558896 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.558852    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfmr2\" (UniqueName: \"kubernetes.io/projected/7ce9cece-477a-4b17-92cd-dd8a1869d000-kube-api-access-qfmr2\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:46.558896 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:46.558886    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce9cece-477a-4b17-92cd-dd8a1869d000-plugins-config-volume\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:47.042444 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.042404    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ce9cece-477a-4b17-92cd-dd8a1869d000" containerID="2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2" exitCode=0
Sep 08 14:00:47.042597 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.042466    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"
Sep 08 14:00:47.042597 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.042492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w" event={"ID":"7ce9cece-477a-4b17-92cd-dd8a1869d000","Type":"ContainerDied","Data":"2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2"}
Sep 08 14:00:47.042597 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.042548    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w" event={"ID":"7ce9cece-477a-4b17-92cd-dd8a1869d000","Type":"ContainerDied","Data":"9ec1c3b0a04949ae41f19e668458211d0eed3bb7f3c3d718f9ada3de1437f906"}
Sep 08 14:00:47.042597 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.042571    2561 scope.go:117] "RemoveContainer" containerID="2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2"
Sep 08 14:00:47.051708 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.051689    2561 scope.go:117] "RemoveContainer" containerID="2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2"
Sep 08 14:00:47.051976 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:47.051958    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2\": container with ID starting with 2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2 not found: ID does not exist" containerID="2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2"
Sep 08 14:00:47.052043 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.051988    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2"} err="failed to get container status \"2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2\": rpc error: code = NotFound desc = could not find container \"2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2\": container with ID starting with 2b6968a034eb9f53d1de2541681ee1afc55ac98c3e773a37e996a9e08ad4c7a2 not found: ID does not exist"
Sep 08 14:00:47.063461 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.063431    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"]
Sep 08 14:00:47.066762 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:47.066741    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-905fec-7cb5bbd67d-lcf8w"]
Sep 08 14:00:48.244842 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:48.244808    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ce9cece-477a-4b17-92cd-dd8a1869d000" path="/var/lib/kubelet/pods/7ce9cece-477a-4b17-92cd-dd8a1869d000/volumes"
Sep 08 14:00:48.245305 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:48.245141    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c" path="/var/lib/kubelet/pods/a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c/volumes"
Sep 08 14:00:53.009552 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:53.009513    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"]
Sep 08 14:00:53.010038 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:53.009817    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy" containerID="cri-o://b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15" gracePeriod=30
Sep 08 14:00:58.253528 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.253503    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 14:00:58.344736 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344651    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-data\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.344736 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lclb\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-kube-api-access-6lclb\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.344736 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344715    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee51e98-fe87-4857-a6bf-9094afefede5-istiod-ca-cert\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.344736 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344739    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-certs\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.345053 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344754    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-token\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.345053 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-credential-socket\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.345053 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-envoy\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.345053 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344838    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-podinfo\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.345053 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-socket\") pod \"7ee51e98-fe87-4857-a6bf-9094afefede5\" (UID: \"7ee51e98-fe87-4857-a6bf-9094afefede5\") "
Sep 08 14:00:58.345053 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.344956    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:58.345363 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.345099    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-data\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:58.345363 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.345168    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:58.345363 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.345271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee51e98-fe87-4857-a6bf-9094afefede5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:58.345363 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.345265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:58.345363 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.345334    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:58.347111 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.347077    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:58.347219 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.347126    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 14:00:58.347407 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.347361    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:58.347491 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.347404    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-kube-api-access-6lclb" (OuterVolumeSpecName: "kube-api-access-6lclb") pod "7ee51e98-fe87-4857-a6bf-9094afefede5" (UID: "7ee51e98-fe87-4857-a6bf-9094afefede5"). InnerVolumeSpecName "kube-api-access-6lclb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:58.445797 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.445756    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:58.445797 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.445791    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-token\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:58.445797 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.445803    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-credential-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:58.446026 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.445816    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-envoy\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:58.446026 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.445828    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee51e98-fe87-4857-a6bf-9094afefede5-istio-podinfo\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:58.446026 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.445838    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee51e98-fe87-4857-a6bf-9094afefede5-workload-socket\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:58.446026 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.445850    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lclb\" (UniqueName: \"kubernetes.io/projected/7ee51e98-fe87-4857-a6bf-9094afefede5-kube-api-access-6lclb\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:58.446026 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:58.445861    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee51e98-fe87-4857-a6bf-9094afefede5-istiod-ca-cert\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:00:59.098738 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.098700    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerID="b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15" exitCode=0
Sep 08 14:00:59.098738 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.098740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" event={"ID":"7ee51e98-fe87-4857-a6bf-9094afefede5","Type":"ContainerDied","Data":"b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15"}
Sep 08 14:00:59.098950 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.098746    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"
Sep 08 14:00:59.098950 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.098781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn" event={"ID":"7ee51e98-fe87-4857-a6bf-9094afefede5","Type":"ContainerDied","Data":"5d1914f2da68bf00db40bbe152a011669158938e4c2c96c0a6ecbc9aabe53be0"}
Sep 08 14:00:59.098950 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.098803    2561 scope.go:117] "RemoveContainer" containerID="b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15"
Sep 08 14:00:59.110130 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.110112    2561 scope.go:117] "RemoveContainer" containerID="35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522"
Sep 08 14:00:59.117972 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.117949    2561 scope.go:117] "RemoveContainer" containerID="b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15"
Sep 08 14:00:59.118254 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:59.118234    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15\": container with ID starting with b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15 not found: ID does not exist" containerID="b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15"
Sep 08 14:00:59.118301 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.118263    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15"} err="failed to get container status \"b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15\": rpc error: code = NotFound desc = could not find container \"b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15\": container with ID starting with b6456620739e1f3175890a086a1fe83469662f18080e9a742baed72ca8a8ae15 not found: ID does not exist"
Sep 08 14:00:59.118301 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.118282    2561 scope.go:117] "RemoveContainer" containerID="35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522"
Sep 08 14:00:59.118578 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:00:59.118559    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522\": container with ID starting with 35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522 not found: ID does not exist" containerID="35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522"
Sep 08 14:00:59.118653 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.118584    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522"} err="failed to get container status \"35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522\": rpc error: code = NotFound desc = could not find container \"35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522\": container with ID starting with 35f1265673df0184a1a9943e366092924e70df108756937e4d9027edcf08e522 not found: ID does not exist"
Sep 08 14:00:59.121067 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.121043    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"]
Sep 08 14:00:59.126834 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:00:59.126813    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-905fec-openshift-default-578f474f5d-trxdn"]
Sep 08 14:01:00.244694 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:01:00.244660    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" path="/var/lib/kubelet/pods/7ee51e98-fe87-4857-a6bf-9094afefede5/volumes"
Sep 08 14:01:12.554734 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:01:12.554651    2561 scope.go:117] "RemoveContainer" containerID="9ddfdd2d583f134614e975b0429802b36bd9c1063413f9b67627177778bf40bf"
Sep 08 14:02:12.591765 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:02:12.591731    2561 scope.go:117] "RemoveContainer" containerID="4e7bec401fc1a9b0c790e4b28700c96fbcca53ff68a7eff5ce8f265580158744"
Sep 08 14:05:12.621170 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:12.621134    2561 scope.go:117] "RemoveContainer" containerID="cc29151878e19225932b5904b4b2bd59aafccaca9d8ff3f4fa947f7337621532"
Sep 08 14:05:35.538343 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538260    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"]
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538576    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ce9cece-477a-4b17-92cd-dd8a1869d000" containerName="payload-pre-processing"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538590    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ce9cece-477a-4b17-92cd-dd8a1869d000" containerName="payload-pre-processing"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538603    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538611    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538620    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538627    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538640    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538646    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538659    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c" containerName="revoke-keys"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538665    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c" containerName="revoke-keys"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538679    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538688    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe68d607-a87a-4aad-a976-94439d5d793f" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538734    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538744    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538751    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ce9cece-477a-4b17-92cd-dd8a1869d000" containerName="payload-pre-processing"
Sep 08 14:05:35.538830 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.538760    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a519ffe4-1ea5-4a5b-bcfb-4d9f141ce31c" containerName="revoke-keys"
Sep 08 14:05:35.541651 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.541614    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.544620 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.544597    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9140602a-kserve-self-signed-certs\""
Sep 08 14:05:35.549847 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.549823    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"]
Sep 08 14:05:35.642607 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.642553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-dshm\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.642607 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.642608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4hs\" (UniqueName: \"kubernetes.io/projected/3902c1bd-61ad-4bec-a583-74bddf607097-kube-api-access-8c4hs\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.642834 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.642669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3902c1bd-61ad-4bec-a583-74bddf607097-tls-certs\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.642834 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.642708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-model-cache\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.642834 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.642747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-home\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.642834 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.642769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-tmp-dir\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.743785 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.743736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-dshm\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.743955 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.743788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4hs\" (UniqueName: \"kubernetes.io/projected/3902c1bd-61ad-4bec-a583-74bddf607097-kube-api-access-8c4hs\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.743955 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.743821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3902c1bd-61ad-4bec-a583-74bddf607097-tls-certs\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.743955 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.743847    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-model-cache\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.743955 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.743893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-home\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.743955 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.743925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-tmp-dir\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.744267 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.744245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-model-cache\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.744336 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.744286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-home\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.744403 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.744332    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-tmp-dir\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.746098 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.746080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-dshm\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.746285 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.746266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3902c1bd-61ad-4bec-a583-74bddf607097-tls-certs\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.752461 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.752433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4hs\" (UniqueName: \"kubernetes.io/projected/3902c1bd-61ad-4bec-a583-74bddf607097-kube-api-access-8c4hs\") pod \"e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") " pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:35.851768 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:35.851674    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:36.122473 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:36.122390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"]
Sep 08 14:05:36.126251 ip-10-0-130-208 kubenswrapper[2561]: W0908 14:05:36.126224    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3902c1bd_61ad_4bec_a583_74bddf607097.slice/crio-2f5b42c6c308dad7cbd27c829ac3b219293aa3b96ed083e3ec9f9558b771875b WatchSource:0}: Error finding container 2f5b42c6c308dad7cbd27c829ac3b219293aa3b96ed083e3ec9f9558b771875b: Status 404 returned error can't find the container with id 2f5b42c6c308dad7cbd27c829ac3b219293aa3b96ed083e3ec9f9558b771875b
Sep 08 14:05:36.273113 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:36.273067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx" event={"ID":"3902c1bd-61ad-4bec-a583-74bddf607097","Type":"ContainerStarted","Data":"7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d"}
Sep 08 14:05:36.273113 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:36.273117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx" event={"ID":"3902c1bd-61ad-4bec-a583-74bddf607097","Type":"ContainerStarted","Data":"2f5b42c6c308dad7cbd27c829ac3b219293aa3b96ed083e3ec9f9558b771875b"}
Sep 08 14:05:36.292915 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:36.292862    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx" podStartSLOduration=1.292839107 podStartE2EDuration="1.292839107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:05:36.291756309 +0000 UTC m=+2244.642742258" watchObservedRunningTime="2026-09-08 14:05:36.292839107 +0000 UTC m=+2244.643825054"
Sep 08 14:05:45.852446 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:45.852409    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:45.852913 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:45.852466    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:45.865020 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:45.864994    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:46.326757 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:46.326728    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:05:47.034252 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.034214    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"]
Sep 08 14:05:47.034703 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.034528    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy"
Sep 08 14:05:47.034703 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.034541    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy"
Sep 08 14:05:47.034703 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.034602    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50bf45d-2f90-4d53-bb87-6b8d98246502" containerName="istio-proxy"
Sep 08 14:05:47.034703 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.034612    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee51e98-fe87-4857-a6bf-9094afefede5" containerName="istio-proxy"
Sep 08 14:05:47.037546 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.037528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.040044 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.040019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9140602a-kserve-self-signed-certs\""
Sep 08 14:05:47.045528 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.045504    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"]
Sep 08 14:05:47.138736 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.138698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-tmp-dir\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.138736 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.138733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntzb\" (UniqueName: \"kubernetes.io/projected/486a121c-686f-4402-82b8-9189c8486173-kube-api-access-fntzb\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.138954 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.138768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-home\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.138954 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.138844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-model-cache\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.138954 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.138893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-dshm\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.138954 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.138914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a121c-686f-4402-82b8-9189c8486173-tls-certs\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.239628 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.239583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-tmp-dir\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.239628 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.239633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntzb\" (UniqueName: \"kubernetes.io/projected/486a121c-686f-4402-82b8-9189c8486173-kube-api-access-fntzb\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.239900 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.239679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-home\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.239900 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.239699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-model-cache\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.239900 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.239754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-dshm\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.239900 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.239779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a121c-686f-4402-82b8-9189c8486173-tls-certs\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.240122 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.240052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-home\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.240122 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.240085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-tmp-dir\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.240233 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.240156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-model-cache\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.242090 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.242070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-dshm\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.242375 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.242356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a121c-686f-4402-82b8-9189c8486173-tls-certs\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.247625 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.247600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntzb\" (UniqueName: \"kubernetes.io/projected/486a121c-686f-4402-82b8-9189c8486173-kube-api-access-fntzb\") pod \"e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") " pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.347372 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.347276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:47.616435 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:47.616405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"]
Sep 08 14:05:47.618723 ip-10-0-130-208 kubenswrapper[2561]: W0908 14:05:47.618693    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod486a121c_686f_4402_82b8_9189c8486173.slice/crio-f1ed8dcd04376d956b04a5035461c38de5ced6181f16b4ca8e3dccdbd0ebd36f WatchSource:0}: Error finding container f1ed8dcd04376d956b04a5035461c38de5ced6181f16b4ca8e3dccdbd0ebd36f: Status 404 returned error can't find the container with id f1ed8dcd04376d956b04a5035461c38de5ced6181f16b4ca8e3dccdbd0ebd36f
Sep 08 14:05:48.324958 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:48.324917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48" event={"ID":"486a121c-686f-4402-82b8-9189c8486173","Type":"ContainerStarted","Data":"1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a"}
Sep 08 14:05:48.325358 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:48.324968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48" event={"ID":"486a121c-686f-4402-82b8-9189c8486173","Type":"ContainerStarted","Data":"f1ed8dcd04376d956b04a5035461c38de5ced6181f16b4ca8e3dccdbd0ebd36f"}
Sep 08 14:05:48.343049 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:48.342973    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48" podStartSLOduration=1.342957203 podStartE2EDuration="1.342957203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:05:48.342241225 +0000 UTC m=+2256.693227196" watchObservedRunningTime="2026-09-08 14:05:48.342957203 +0000 UTC m=+2256.693943162"
Sep 08 14:05:57.348288 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:57.348236    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:57.348288 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:57.348300    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:57.361683 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:57.361656    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:05:57.373688 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:05:57.373662    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:06:02.741593 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:02.741556    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"]
Sep 08 14:06:04.385694 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:04.385662    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"]
Sep 08 14:06:12.638639 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:12.638606    2561 scope.go:117] "RemoveContainer" containerID="95293d3dbde51838781929a2a86e1df16bfbd0efd753c4f0a6d37eaa60bf9929"
Sep 08 14:06:12.646422 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:12.646398    2561 scope.go:117] "RemoveContainer" containerID="d6e1467f3625e560e2ce5823b06fa5c6171f48dba7b8623ddf976d27a40fdc24"
Sep 08 14:06:17.761643 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:17.761594    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx" podUID="3902c1bd-61ad-4bec-a583-74bddf607097" containerName="main" containerID="cri-o://7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d" gracePeriod=45
Sep 08 14:06:18.002212 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.002183    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:06:18.099666 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.099574    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3902c1bd-61ad-4bec-a583-74bddf607097-tls-certs\") pod \"3902c1bd-61ad-4bec-a583-74bddf607097\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") "
Sep 08 14:06:18.099666 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.099619    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-tmp-dir\") pod \"3902c1bd-61ad-4bec-a583-74bddf607097\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") "
Sep 08 14:06:18.099666 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.099648    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-dshm\") pod \"3902c1bd-61ad-4bec-a583-74bddf607097\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") "
Sep 08 14:06:18.099959 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.099696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c4hs\" (UniqueName: \"kubernetes.io/projected/3902c1bd-61ad-4bec-a583-74bddf607097-kube-api-access-8c4hs\") pod \"3902c1bd-61ad-4bec-a583-74bddf607097\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") "
Sep 08 14:06:18.099959 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.099723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-model-cache\") pod \"3902c1bd-61ad-4bec-a583-74bddf607097\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") "
Sep 08 14:06:18.099959 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.099771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-home\") pod \"3902c1bd-61ad-4bec-a583-74bddf607097\" (UID: \"3902c1bd-61ad-4bec-a583-74bddf607097\") "
Sep 08 14:06:18.099959 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.099934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3902c1bd-61ad-4bec-a583-74bddf607097" (UID: "3902c1bd-61ad-4bec-a583-74bddf607097"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:18.100150 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.099982    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-model-cache" (OuterVolumeSpecName: "model-cache") pod "3902c1bd-61ad-4bec-a583-74bddf607097" (UID: "3902c1bd-61ad-4bec-a583-74bddf607097"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:18.100150 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.100027    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:18.100150 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.100065    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-home" (OuterVolumeSpecName: "home") pod "3902c1bd-61ad-4bec-a583-74bddf607097" (UID: "3902c1bd-61ad-4bec-a583-74bddf607097"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:18.101853 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.101820    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-dshm" (OuterVolumeSpecName: "dshm") pod "3902c1bd-61ad-4bec-a583-74bddf607097" (UID: "3902c1bd-61ad-4bec-a583-74bddf607097"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:18.101971 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.101850    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3902c1bd-61ad-4bec-a583-74bddf607097-kube-api-access-8c4hs" (OuterVolumeSpecName: "kube-api-access-8c4hs") pod "3902c1bd-61ad-4bec-a583-74bddf607097" (UID: "3902c1bd-61ad-4bec-a583-74bddf607097"). InnerVolumeSpecName "kube-api-access-8c4hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:06:18.101971 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.101880    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3902c1bd-61ad-4bec-a583-74bddf607097-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3902c1bd-61ad-4bec-a583-74bddf607097" (UID: "3902c1bd-61ad-4bec-a583-74bddf607097"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:06:18.201272 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.201234    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3902c1bd-61ad-4bec-a583-74bddf607097-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:18.201272 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.201265    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:18.201272 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.201275    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4hs\" (UniqueName: \"kubernetes.io/projected/3902c1bd-61ad-4bec-a583-74bddf607097-kube-api-access-8c4hs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:18.201554 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.201285    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:18.201554 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.201294    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3902c1bd-61ad-4bec-a583-74bddf607097-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:18.453096 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.453063    2561 generic.go:356] "Generic (PLEG): container finished" podID="3902c1bd-61ad-4bec-a583-74bddf607097" containerID="7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d" exitCode=0
Sep 08 14:06:18.453274 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.453108    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"
Sep 08 14:06:18.453274 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.453152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx" event={"ID":"3902c1bd-61ad-4bec-a583-74bddf607097","Type":"ContainerDied","Data":"7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d"}
Sep 08 14:06:18.453274 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.453201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx" event={"ID":"3902c1bd-61ad-4bec-a583-74bddf607097","Type":"ContainerDied","Data":"2f5b42c6c308dad7cbd27c829ac3b219293aa3b96ed083e3ec9f9558b771875b"}
Sep 08 14:06:18.453274 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.453224    2561 scope.go:117] "RemoveContainer" containerID="7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d"
Sep 08 14:06:18.461219 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.461198    2561 scope.go:117] "RemoveContainer" containerID="7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d"
Sep 08 14:06:18.461579 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:06:18.461561    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d\": container with ID starting with 7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d not found: ID does not exist" containerID="7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d"
Sep 08 14:06:18.461638 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.461589    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d"} err="failed to get container status \"7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d\": rpc error: code = NotFound desc = could not find container \"7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d\": container with ID starting with 7c9f49c47597ad5a7c4cdf50e6ac3f5d366644f9d00f931470aa7fe9b4fca05d not found: ID does not exist"
Sep 08 14:06:18.470172 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.470142    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"]
Sep 08 14:06:18.475032 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:18.475006    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9140602a-kserve-6b9cb5758c-29xpx"]
Sep 08 14:06:19.406116 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.406072    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48" podUID="486a121c-686f-4402-82b8-9189c8486173" containerName="main" containerID="cri-o://1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a" gracePeriod=45
Sep 08 14:06:19.642345 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.642321    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:06:19.817479 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817442    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-dshm\") pod \"486a121c-686f-4402-82b8-9189c8486173\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") "
Sep 08 14:06:19.817479 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817483    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a121c-686f-4402-82b8-9189c8486173-tls-certs\") pod \"486a121c-686f-4402-82b8-9189c8486173\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") "
Sep 08 14:06:19.817734 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817505    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fntzb\" (UniqueName: \"kubernetes.io/projected/486a121c-686f-4402-82b8-9189c8486173-kube-api-access-fntzb\") pod \"486a121c-686f-4402-82b8-9189c8486173\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") "
Sep 08 14:06:19.817734 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817543    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-model-cache\") pod \"486a121c-686f-4402-82b8-9189c8486173\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") "
Sep 08 14:06:19.817734 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817619    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-tmp-dir\") pod \"486a121c-686f-4402-82b8-9189c8486173\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") "
Sep 08 14:06:19.817734 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817643    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-home\") pod \"486a121c-686f-4402-82b8-9189c8486173\" (UID: \"486a121c-686f-4402-82b8-9189c8486173\") "
Sep 08 14:06:19.818005 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817895    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "486a121c-686f-4402-82b8-9189c8486173" (UID: "486a121c-686f-4402-82b8-9189c8486173"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:19.818005 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817933    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-model-cache" (OuterVolumeSpecName: "model-cache") pod "486a121c-686f-4402-82b8-9189c8486173" (UID: "486a121c-686f-4402-82b8-9189c8486173"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:19.818005 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.817983    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-home" (OuterVolumeSpecName: "home") pod "486a121c-686f-4402-82b8-9189c8486173" (UID: "486a121c-686f-4402-82b8-9189c8486173"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:19.819777 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.819753    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486a121c-686f-4402-82b8-9189c8486173-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "486a121c-686f-4402-82b8-9189c8486173" (UID: "486a121c-686f-4402-82b8-9189c8486173"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:06:19.819777 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.819765    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-dshm" (OuterVolumeSpecName: "dshm") pod "486a121c-686f-4402-82b8-9189c8486173" (UID: "486a121c-686f-4402-82b8-9189c8486173"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:06:19.819777 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.819769    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486a121c-686f-4402-82b8-9189c8486173-kube-api-access-fntzb" (OuterVolumeSpecName: "kube-api-access-fntzb") pod "486a121c-686f-4402-82b8-9189c8486173" (UID: "486a121c-686f-4402-82b8-9189c8486173"). InnerVolumeSpecName "kube-api-access-fntzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:06:19.919148 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.919103    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-tmp-dir\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:19.919148 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.919139    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-home\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:19.919148 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.919151    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-dshm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:19.919420 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.919161    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486a121c-686f-4402-82b8-9189c8486173-tls-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:19.919420 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.919173    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fntzb\" (UniqueName: \"kubernetes.io/projected/486a121c-686f-4402-82b8-9189c8486173-kube-api-access-fntzb\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:19.919420 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:19.919185    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486a121c-686f-4402-82b8-9189c8486173-model-cache\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:20.244723 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.244687    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3902c1bd-61ad-4bec-a583-74bddf607097" path="/var/lib/kubelet/pods/3902c1bd-61ad-4bec-a583-74bddf607097/volumes"
Sep 08 14:06:20.465326 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.465291    2561 generic.go:356] "Generic (PLEG): container finished" podID="486a121c-686f-4402-82b8-9189c8486173" containerID="1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a" exitCode=0
Sep 08 14:06:20.465326 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.465329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48" event={"ID":"486a121c-686f-4402-82b8-9189c8486173","Type":"ContainerDied","Data":"1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a"}
Sep 08 14:06:20.465886 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.465355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48" event={"ID":"486a121c-686f-4402-82b8-9189c8486173","Type":"ContainerDied","Data":"f1ed8dcd04376d956b04a5035461c38de5ced6181f16b4ca8e3dccdbd0ebd36f"}
Sep 08 14:06:20.465886 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.465358    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"
Sep 08 14:06:20.465886 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.465372    2561 scope.go:117] "RemoveContainer" containerID="1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a"
Sep 08 14:06:20.475732 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.475695    2561 scope.go:117] "RemoveContainer" containerID="1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a"
Sep 08 14:06:20.476050 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:06:20.476028    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a\": container with ID starting with 1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a not found: ID does not exist" containerID="1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a"
Sep 08 14:06:20.476128 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.476055    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a"} err="failed to get container status \"1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a\": rpc error: code = NotFound desc = could not find container \"1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a\": container with ID starting with 1cd3928214ad05836f2cc0967dd5bc5433bcf26bda135f94e5da941af414051a not found: ID does not exist"
Sep 08 14:06:20.482988 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.482936    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"]
Sep 08 14:06:20.484157 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:20.484135    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9140602a-kserve-74b6488dbc-ttp48"]
Sep 08 14:06:22.244660 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:22.244627    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486a121c-686f-4402-82b8-9189c8486173" path="/var/lib/kubelet/pods/486a121c-686f-4402-82b8-9189c8486173/volumes"
Sep 08 14:06:24.142443 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.142408    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"]
Sep 08 14:06:24.142824 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.142673    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486a121c-686f-4402-82b8-9189c8486173" containerName="main"
Sep 08 14:06:24.142824 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.142683    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="486a121c-686f-4402-82b8-9189c8486173" containerName="main"
Sep 08 14:06:24.142824 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.142697    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3902c1bd-61ad-4bec-a583-74bddf607097" containerName="main"
Sep 08 14:06:24.142824 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.142702    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3902c1bd-61ad-4bec-a583-74bddf607097" containerName="main"
Sep 08 14:06:24.142824 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.142752    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3902c1bd-61ad-4bec-a583-74bddf607097" containerName="main"
Sep 08 14:06:24.142824 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.142762    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="486a121c-686f-4402-82b8-9189c8486173" containerName="main"
Sep 08 14:06:24.146832 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.146812    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.150624 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.150594    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fzrks\""
Sep 08 14:06:24.150781 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.150650    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 14:06:24.150781 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.150695    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 14:06:24.150781 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.150722    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 14:06:24.150781 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.150695    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 14:06:24.153198 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.153171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-maas-api-tls\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.153421 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.153398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ddm\" (UniqueName: \"kubernetes.io/projected/3945e7d2-5bfc-4dfe-8df6-18a47e749856-kube-api-access-85ddm\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.153570 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.153545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-metrics-certs\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.153673 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.153656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3945e7d2-5bfc-4dfe-8df6-18a47e749856-openshift-service-ca\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.155102 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.155079    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"]
Sep 08 14:06:24.254612 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.254577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3945e7d2-5bfc-4dfe-8df6-18a47e749856-openshift-service-ca\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.254820 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.254618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-maas-api-tls\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.254820 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.254652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ddm\" (UniqueName: \"kubernetes.io/projected/3945e7d2-5bfc-4dfe-8df6-18a47e749856-kube-api-access-85ddm\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.254820 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.254699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-metrics-certs\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.255216 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.255193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3945e7d2-5bfc-4dfe-8df6-18a47e749856-openshift-service-ca\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.257095 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.257071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-metrics-certs\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.257192 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.257109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-maas-api-tls\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.262708 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.262687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ddm\" (UniqueName: \"kubernetes.io/projected/3945e7d2-5bfc-4dfe-8df6-18a47e749856-kube-api-access-85ddm\") pod \"maas-api-84d9df9944-v7pkn\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") " pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.458754 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.458716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:24.734734 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.734709    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"]
Sep 08 14:06:24.737094 ip-10-0-130-208 kubenswrapper[2561]: W0908 14:06:24.737063    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3945e7d2_5bfc_4dfe_8df6_18a47e749856.slice/crio-b9be1c423f9a3476675feeb6b98db25ebe126131571713faca2d011c651007f4 WatchSource:0}: Error finding container b9be1c423f9a3476675feeb6b98db25ebe126131571713faca2d011c651007f4: Status 404 returned error can't find the container with id b9be1c423f9a3476675feeb6b98db25ebe126131571713faca2d011c651007f4
Sep 08 14:06:24.738410 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:24.738391    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:06:25.488704 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:25.488658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn" event={"ID":"3945e7d2-5bfc-4dfe-8df6-18a47e749856","Type":"ContainerStarted","Data":"b9be1c423f9a3476675feeb6b98db25ebe126131571713faca2d011c651007f4"}
Sep 08 14:06:27.498467 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:27.498431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn" event={"ID":"3945e7d2-5bfc-4dfe-8df6-18a47e749856","Type":"ContainerStarted","Data":"d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f"}
Sep 08 14:06:27.498888 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:27.498568    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:27.522402 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:27.519277    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn" podStartSLOduration=1.772417395 podStartE2EDuration="3.519254907s" totalImagesPullingTime="1.746837512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:06:27.517692126 +0000 UTC m=+2295.868678077" watchObservedRunningTime="2026-09-08 14:06:27.519254907 +0000 UTC m=+2295.870240856"
Sep 08 14:06:31.860038 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.860001    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"]
Sep 08 14:06:31.860451 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.860197    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn" podUID="3945e7d2-5bfc-4dfe-8df6-18a47e749856" containerName="maas-api" containerID="cri-o://d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f" gracePeriod=30
Sep 08 14:06:31.865271 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.865248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:31.896709 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.896673    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"]
Sep 08 14:06:31.899935 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.899918    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:31.904691 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.904642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252e34d4-38d1-46d0-8c9a-afb5bac9a313-metrics-certs\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:31.904848 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.904721    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6s7\" (UniqueName: \"kubernetes.io/projected/252e34d4-38d1-46d0-8c9a-afb5bac9a313-kube-api-access-gd6s7\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:31.904848 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.904776    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252e34d4-38d1-46d0-8c9a-afb5bac9a313-maas-api-tls\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:31.904848 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.904793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252e34d4-38d1-46d0-8c9a-afb5bac9a313-openshift-service-ca\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:31.908510 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:31.908484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"]
Sep 08 14:06:32.005786 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.005748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252e34d4-38d1-46d0-8c9a-afb5bac9a313-maas-api-tls\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.005786 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.005785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252e34d4-38d1-46d0-8c9a-afb5bac9a313-openshift-service-ca\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.006047 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.005843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252e34d4-38d1-46d0-8c9a-afb5bac9a313-metrics-certs\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.006047 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.005878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6s7\" (UniqueName: \"kubernetes.io/projected/252e34d4-38d1-46d0-8c9a-afb5bac9a313-kube-api-access-gd6s7\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.006654 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.006630    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252e34d4-38d1-46d0-8c9a-afb5bac9a313-openshift-service-ca\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.008478 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.008450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252e34d4-38d1-46d0-8c9a-afb5bac9a313-metrics-certs\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.008614 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.008583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252e34d4-38d1-46d0-8c9a-afb5bac9a313-maas-api-tls\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.017942 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.017915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6s7\" (UniqueName: \"kubernetes.io/projected/252e34d4-38d1-46d0-8c9a-afb5bac9a313-kube-api-access-gd6s7\") pod \"maas-api-86dc7466b4-vxpfr\" (UID: \"252e34d4-38d1-46d0-8c9a-afb5bac9a313\") " pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.100697 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.100666    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:32.107033 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.107016    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-metrics-certs\") pod \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") "
Sep 08 14:06:32.107098 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.107061    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85ddm\" (UniqueName: \"kubernetes.io/projected/3945e7d2-5bfc-4dfe-8df6-18a47e749856-kube-api-access-85ddm\") pod \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") "
Sep 08 14:06:32.107098 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.107090    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3945e7d2-5bfc-4dfe-8df6-18a47e749856-openshift-service-ca\") pod \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") "
Sep 08 14:06:32.107167 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.107113    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-maas-api-tls\") pod \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\" (UID: \"3945e7d2-5bfc-4dfe-8df6-18a47e749856\") "
Sep 08 14:06:32.107506 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.107475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3945e7d2-5bfc-4dfe-8df6-18a47e749856-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3945e7d2-5bfc-4dfe-8df6-18a47e749856" (UID: "3945e7d2-5bfc-4dfe-8df6-18a47e749856"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:06:32.109291 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.109268    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3945e7d2-5bfc-4dfe-8df6-18a47e749856-kube-api-access-85ddm" (OuterVolumeSpecName: "kube-api-access-85ddm") pod "3945e7d2-5bfc-4dfe-8df6-18a47e749856" (UID: "3945e7d2-5bfc-4dfe-8df6-18a47e749856"). InnerVolumeSpecName "kube-api-access-85ddm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:06:32.109356 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.109295    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3945e7d2-5bfc-4dfe-8df6-18a47e749856" (UID: "3945e7d2-5bfc-4dfe-8df6-18a47e749856"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:06:32.109356 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.109320    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3945e7d2-5bfc-4dfe-8df6-18a47e749856" (UID: "3945e7d2-5bfc-4dfe-8df6-18a47e749856"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:06:32.208655 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.208617    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ddm\" (UniqueName: \"kubernetes.io/projected/3945e7d2-5bfc-4dfe-8df6-18a47e749856-kube-api-access-85ddm\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:32.208655 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.208651    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3945e7d2-5bfc-4dfe-8df6-18a47e749856-openshift-service-ca\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:32.208655 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.208661    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-maas-api-tls\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:32.208904 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.208669    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3945e7d2-5bfc-4dfe-8df6-18a47e749856-metrics-certs\") on node \"ip-10-0-130-208.ec2.internal\" DevicePath \"\""
Sep 08 14:06:32.241681 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.241636    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:32.513354 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.513328    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"]
Sep 08 14:06:32.515791 ip-10-0-130-208 kubenswrapper[2561]: W0908 14:06:32.515749    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252e34d4_38d1_46d0_8c9a_afb5bac9a313.slice/crio-e1923df2a8273ec0f766f4519b3da0574fb2843da41d9f00d6caa81062748933 WatchSource:0}: Error finding container e1923df2a8273ec0f766f4519b3da0574fb2843da41d9f00d6caa81062748933: Status 404 returned error can't find the container with id e1923df2a8273ec0f766f4519b3da0574fb2843da41d9f00d6caa81062748933
Sep 08 14:06:32.525643 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.525609    2561 generic.go:356] "Generic (PLEG): container finished" podID="3945e7d2-5bfc-4dfe-8df6-18a47e749856" containerID="d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f" exitCode=0
Sep 08 14:06:32.525769 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.525669    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"
Sep 08 14:06:32.525769 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.525671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn" event={"ID":"3945e7d2-5bfc-4dfe-8df6-18a47e749856","Type":"ContainerDied","Data":"d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f"}
Sep 08 14:06:32.525857 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.525808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn" event={"ID":"3945e7d2-5bfc-4dfe-8df6-18a47e749856","Type":"ContainerDied","Data":"b9be1c423f9a3476675feeb6b98db25ebe126131571713faca2d011c651007f4"}
Sep 08 14:06:32.525857 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.525839    2561 scope.go:117] "RemoveContainer" containerID="d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f"
Sep 08 14:06:32.534404 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.534370    2561 scope.go:117] "RemoveContainer" containerID="d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f"
Sep 08 14:06:32.534683 ip-10-0-130-208 kubenswrapper[2561]: E0908 14:06:32.534662    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f\": container with ID starting with d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f not found: ID does not exist" containerID="d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f"
Sep 08 14:06:32.534741 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.534692    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f"} err="failed to get container status \"d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f\": rpc error: code = NotFound desc = could not find container \"d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f\": container with ID starting with d03972590155ae50a4234e1814e6cf05af449ceb5b18abc4313f5386aa61411f not found: ID does not exist"
Sep 08 14:06:32.542333 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.542306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"]
Sep 08 14:06:32.546008 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:32.545985    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84d9df9944-v7pkn"]
Sep 08 14:06:33.531448 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:33.531405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr" event={"ID":"252e34d4-38d1-46d0-8c9a-afb5bac9a313","Type":"ContainerStarted","Data":"df4e1aa66c3f66ee7dd57a47d104a702344e85a9df80337bf6ae986d582dc1f8"}
Sep 08 14:06:33.531896 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:33.531470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr" event={"ID":"252e34d4-38d1-46d0-8c9a-afb5bac9a313","Type":"ContainerStarted","Data":"e1923df2a8273ec0f766f4519b3da0574fb2843da41d9f00d6caa81062748933"}
Sep 08 14:06:33.531896 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:33.531498    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:33.550460 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:33.550411    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr" podStartSLOduration=2.136699893 podStartE2EDuration="2.55039485s" totalImagesPullingTime="413.694957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:06:33.548819483 +0000 UTC m=+2301.899805431" watchObservedRunningTime="2026-09-08 14:06:33.55039485 +0000 UTC m=+2301.901380789"
Sep 08 14:06:34.245064 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:34.245028    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3945e7d2-5bfc-4dfe-8df6-18a47e749856" path="/var/lib/kubelet/pods/3945e7d2-5bfc-4dfe-8df6-18a47e749856/volumes"
Sep 08 14:06:39.541252 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:39.541222    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86dc7466b4-vxpfr"
Sep 08 14:06:57.521344 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:06:57.521306    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dcccdf9cc-zw28h_8567f5f6-08b5-43ea-b35e-9b77dea07c42/authorino/0.log"
Sep 08 14:07:03.649751 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:03.649721    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p58pj_a1adf55b-2b9e-4a5b-a060-09475b42475c/server/0.log"
Sep 08 14:07:03.774246 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:03.774220    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-75wrn_0f78383c-61b1-4516-afce-f6dd94051218/manager/0.log"
Sep 08 14:07:04.022773 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:04.022741    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54685d959b-vzcc8_069ade53-d205-44e5-9eef-7d7e5cef93a8/copy-manifests/0.log"
Sep 08 14:07:04.029157 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:04.029134    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54685d959b-vzcc8_069ade53-d205-44e5-9eef-7d7e5cef93a8/manager/0.log"
Sep 08 14:07:04.654227 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:04.654156    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86dc7466b4-vxpfr_252e34d4-38d1-46d0-8c9a-afb5bac9a313/maas-api/0.log"
Sep 08 14:07:05.903567 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:05.903537    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dcccdf9cc-zw28h_8567f5f6-08b5-43ea-b35e-9b77dea07c42/authorino/0.log"
Sep 08 14:07:07.437770 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:07.437742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55db547db7-tslwp_a6b092c6-3ec8-4550-9aea-5354fe082416/kube-auth-proxy/0.log"
Sep 08 14:07:08.498658 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:08.498624    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tnxsx_cad7f373-91df-408b-8fac-4a0a889f5b05/main/0.log"
Sep 08 14:07:09.119142 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:09.119113    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-98ggc_1e197378-390c-46bc-a252-2a62ff4a4999/main/0.log"
Sep 08 14:07:12.666720 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:12.666609    2561 scope.go:117] "RemoveContainer" containerID="3b3e376926b7e01a65fa3d630265cdf8fab08d92d099fb72ee2d31c5f1a06f85"
Sep 08 14:07:17.679518 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:17.679485    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wcdnb_04639862-2bd6-4c98-896e-473ccb2d7bd7/global-pull-secret-syncer/0.log"
Sep 08 14:07:17.759851 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:17.759817    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q7z85_38ef716f-8b3c-4b09-b816-1e59fdefc265/konnectivity-agent/0.log"
Sep 08 14:07:17.832530 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:17.832498    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-208.ec2.internal_9135ca1286983be48fef1a9d4c6b86dc/haproxy/0.log"
Sep 08 14:07:22.343872 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:22.343833    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dcccdf9cc-zw28h_8567f5f6-08b5-43ea-b35e-9b77dea07c42/authorino/0.log"
Sep 08 14:07:24.622588 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:24.622554    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prjvg_bc18798d-e11e-4b96-8e80-14e22e838765/node-exporter/0.log"
Sep 08 14:07:24.646583 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:24.646557    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prjvg_bc18798d-e11e-4b96-8e80-14e22e838765/kube-rbac-proxy/0.log"
Sep 08 14:07:24.674580 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:24.674553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prjvg_bc18798d-e11e-4b96-8e80-14e22e838765/init-textfile/0.log"
Sep 08 14:07:29.182628 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.182589    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"]
Sep 08 14:07:29.183133 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.182880    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3945e7d2-5bfc-4dfe-8df6-18a47e749856" containerName="maas-api"
Sep 08 14:07:29.183133 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.182892    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3945e7d2-5bfc-4dfe-8df6-18a47e749856" containerName="maas-api"
Sep 08 14:07:29.183133 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.182950    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3945e7d2-5bfc-4dfe-8df6-18a47e749856" containerName="maas-api"
Sep 08 14:07:29.185836 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.185818    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.188777 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.188755    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52jtx\"/\"kube-root-ca.crt\""
Sep 08 14:07:29.189603 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.189585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52jtx\"/\"openshift-service-ca.crt\""
Sep 08 14:07:29.189661 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.189611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-52jtx\"/\"default-dockercfg-dxk4h\""
Sep 08 14:07:29.196283 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.196258    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"]
Sep 08 14:07:29.275548 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.275501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-podres\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.275548 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.275550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrmz\" (UniqueName: \"kubernetes.io/projected/7b290777-e1e2-4557-ac86-f6087e0f5f49-kube-api-access-drrmz\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.275793 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.275575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-proc\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.275793 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.275609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-lib-modules\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.275793 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.275653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-sys\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376401 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376337    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-lib-modules\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376576 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-sys\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376576 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-podres\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376576 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrmz\" (UniqueName: \"kubernetes.io/projected/7b290777-e1e2-4557-ac86-f6087e0f5f49-kube-api-access-drrmz\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376576 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-lib-modules\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376576 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376540    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-sys\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376768 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-podres\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376768 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-proc\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.376768 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.376647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b290777-e1e2-4557-ac86-f6087e0f5f49-proc\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.385030 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.385010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrmz\" (UniqueName: \"kubernetes.io/projected/7b290777-e1e2-4557-ac86-f6087e0f5f49-kube-api-access-drrmz\") pod \"perf-node-gather-daemonset-25g45\" (UID: \"7b290777-e1e2-4557-ac86-f6087e0f5f49\") " pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.496076 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.495981    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:29.766428 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.766390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"]
Sep 08 14:07:29.769105 ip-10-0-130-208 kubenswrapper[2561]: W0908 14:07:29.769074    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7b290777_e1e2_4557_ac86_f6087e0f5f49.slice/crio-7df4ceb2f1dfcc6bb3106ba748435db0a1c828deb3e3004eebf171ef704119cd WatchSource:0}: Error finding container 7df4ceb2f1dfcc6bb3106ba748435db0a1c828deb3e3004eebf171ef704119cd: Status 404 returned error can't find the container with id 7df4ceb2f1dfcc6bb3106ba748435db0a1c828deb3e3004eebf171ef704119cd
Sep 08 14:07:29.774759 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:29.774723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45" event={"ID":"7b290777-e1e2-4557-ac86-f6087e0f5f49","Type":"ContainerStarted","Data":"7df4ceb2f1dfcc6bb3106ba748435db0a1c828deb3e3004eebf171ef704119cd"}
Sep 08 14:07:30.238832 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:30.238800    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z4pq8_02d2b805-a009-4c2b-9400-9fedb9394010/dns/0.log"
Sep 08 14:07:30.258877 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:30.258851    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z4pq8_02d2b805-a009-4c2b-9400-9fedb9394010/kube-rbac-proxy/0.log"
Sep 08 14:07:30.311077 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:30.311049    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pl7gt_9f67904d-8e19-44ee-a8ed-9da81a313102/dns-node-resolver/0.log"
Sep 08 14:07:30.780354 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:30.780316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45" event={"ID":"7b290777-e1e2-4557-ac86-f6087e0f5f49","Type":"ContainerStarted","Data":"644aa20ed1d4eb17019fc95b4598831b8c1630e8f0f99c3e569e12f73967a131"}
Sep 08 14:07:30.780578 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:30.780437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:30.799340 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:30.799283    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45" podStartSLOduration=1.799266603 podStartE2EDuration="1.799266603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:07:30.796664918 +0000 UTC m=+2359.147650893" watchObservedRunningTime="2026-09-08 14:07:30.799266603 +0000 UTC m=+2359.150252551"
Sep 08 14:07:30.953600 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:30.953566    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hdfv6_91b8aa20-62a5-4327-ba6f-563710623ce6/node-ca/0.log"
Sep 08 14:07:32.259573 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:32.259544    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55db547db7-tslwp_a6b092c6-3ec8-4550-9aea-5354fe082416/kube-auth-proxy/0.log"
Sep 08 14:07:33.298129 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:33.298086    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l2pgh_5d9c002e-39e8-43b7-9548-f275c98d915e/serve-healthcheck-canary/0.log"
Sep 08 14:07:33.993772 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:33.993740    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zzgcc_c27f9499-cc5a-47ec-849f-838cd3698568/kube-rbac-proxy/0.log"
Sep 08 14:07:34.019686 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:34.019665    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zzgcc_c27f9499-cc5a-47ec-849f-838cd3698568/exporter/0.log"
Sep 08 14:07:34.043308 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:34.043283    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zzgcc_c27f9499-cc5a-47ec-849f-838cd3698568/extractor/0.log"
Sep 08 14:07:36.508554 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:36.508524    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p58pj_a1adf55b-2b9e-4a5b-a060-09475b42475c/server/0.log"
Sep 08 14:07:36.589693 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:36.589660    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-75wrn_0f78383c-61b1-4516-afce-f6dd94051218/manager/0.log"
Sep 08 14:07:36.669960 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:36.669923    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54685d959b-vzcc8_069ade53-d205-44e5-9eef-7d7e5cef93a8/manager/0.log"
Sep 08 14:07:36.692852 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:36.692819    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54685d959b-vzcc8_069ade53-d205-44e5-9eef-7d7e5cef93a8/copy-manifests/0.log"
Sep 08 14:07:36.794831 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:36.794751    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-52jtx/perf-node-gather-daemonset-25g45"
Sep 08 14:07:38.082276 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:38.082197    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8d99975cf-tgbgj_a63233e2-b5f0-471c-b57f-bc5b91441c5f/manager/0.log"
Sep 08 14:07:39.201930 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:39.201898    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55db547db7-tslwp_a6b092c6-3ec8-4550-9aea-5354fe082416/kube-auth-proxy/0.log"
Sep 08 14:07:41.169241 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:41.169202    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55db547db7-tslwp_a6b092c6-3ec8-4550-9aea-5354fe082416/kube-auth-proxy/0.log"
Sep 08 14:07:45.405512 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.405433    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8vsw_f3cc2402-36f3-4342-8d14-e978f9e66436/kube-multus-additional-cni-plugins/0.log"
Sep 08 14:07:45.430779 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.430750    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8vsw_f3cc2402-36f3-4342-8d14-e978f9e66436/egress-router-binary-copy/0.log"
Sep 08 14:07:45.450956 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.450927    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8vsw_f3cc2402-36f3-4342-8d14-e978f9e66436/cni-plugins/0.log"
Sep 08 14:07:45.473102 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.473077    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8vsw_f3cc2402-36f3-4342-8d14-e978f9e66436/bond-cni-plugin/0.log"
Sep 08 14:07:45.495846 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.495817    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8vsw_f3cc2402-36f3-4342-8d14-e978f9e66436/routeoverride-cni/0.log"
Sep 08 14:07:45.517916 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.517892    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8vsw_f3cc2402-36f3-4342-8d14-e978f9e66436/whereabouts-cni-bincopy/0.log"
Sep 08 14:07:45.542430 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.542405    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n8vsw_f3cc2402-36f3-4342-8d14-e978f9e66436/whereabouts-cni/0.log"
Sep 08 14:07:45.790502 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.790474    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rjgp2_10415704-0e6c-43d8-b16f-c5a2ee365b5e/kube-multus/0.log"
Sep 08 14:07:45.916347 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.916310    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zspg2_6404c150-528e-4903-8049-9fa576f8f9ea/network-metrics-daemon/0.log"
Sep 08 14:07:45.935928 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:45.935902    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zspg2_6404c150-528e-4903-8049-9fa576f8f9ea/kube-rbac-proxy/0.log"
Sep 08 14:07:46.807926 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:46.807897    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bjbc_890f71c6-95de-42c3-9790-4d3c7155687d/ovn-controller/0.log"
Sep 08 14:07:46.826021 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:46.825993    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bjbc_890f71c6-95de-42c3-9790-4d3c7155687d/ovn-acl-logging/0.log"
Sep 08 14:07:46.851078 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:46.851039    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bjbc_890f71c6-95de-42c3-9790-4d3c7155687d/kube-rbac-proxy-node/0.log"
Sep 08 14:07:46.872223 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:46.872193    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bjbc_890f71c6-95de-42c3-9790-4d3c7155687d/kube-rbac-proxy-ovn-metrics/0.log"
Sep 08 14:07:46.892619 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:46.892592    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bjbc_890f71c6-95de-42c3-9790-4d3c7155687d/northd/0.log"
Sep 08 14:07:46.913809 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:46.913786    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bjbc_890f71c6-95de-42c3-9790-4d3c7155687d/nbdb/0.log"
Sep 08 14:07:46.936229 ip-10-0-130-208 kubenswrapper[2561]: I0908 14:07:46.936198    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bjbc_890f71c6-95de-42c3-9790-4d3c7155687d/sbdb/0.log"