Sep 11 19:27:17.841633 ip-10-0-136-27 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 19:27:17.841642 ip-10-0-136-27 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 19:27:17.841650 ip-10-0-136-27 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 19:27:17.841863 ip-10-0-136-27 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 19:27:28.074627 ip-10-0-136-27 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 19:27:28.074645 ip-10-0-136-27 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d93dbaccae1943288fe483502766f73e --
Sep 11 19:29:23.214357 ip-10-0-136-27 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 19:29:23.577410 ip-10-0-136-27 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 19:29:23.577410 ip-10-0-136-27 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 19:29:23.577410 ip-10-0-136-27 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 19:29:23.577410 ip-10-0-136-27 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 19:29:23.577410 ip-10-0-136-27 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 19:29:23.579897 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.579819    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 19:29:23.585161 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585146    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:29:23.585161 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585161    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585165    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585168    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585171    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585173    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585176    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585179    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585182    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585184    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585186    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585189    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585192    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585195    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585197    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585200    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585203    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585205    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585208    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585211    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:29:23.585226 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585214    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585216    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585221    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585225    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585228    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585231    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585234    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585238    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585241    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585244    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585247    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585251    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585255    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585258    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585261    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585264    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585267    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585270    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585273    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:29:23.585704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585276    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585279    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585283    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585287    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585290    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585293    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585295    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585298    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585301    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585304    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585307    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585311    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585314    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585317    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585320    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585323    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585325    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585328    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585330    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:29:23.586233 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585333    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585336    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585338    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585341    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585344    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585347    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585349    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585360    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585363    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585366    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585369    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585371    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585374    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585376    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585379    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585382    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585384    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585387    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585389    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585392    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:29:23.586698 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585394    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585397    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585400    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585402    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585404    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585407    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585410    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585413    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585416    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585419    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585422    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585424    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585427    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585429    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585432    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585434    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585437    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585443    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585446    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:29:23.587202 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585448    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.585451    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586201    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586212    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586215    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586221    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586227    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586230    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586233    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586236    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586239    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586242    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586245    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586248    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586251    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586254    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586257    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586260    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586263    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:29:23.587674 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586265    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586268    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586271    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586274    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586276    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586279    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586281    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586284    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586286    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586289    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586292    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586294    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586297    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586300    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586302    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586304    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586308    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586311    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586313    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:29:23.588149 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586316    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586319    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586321    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586324    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586326    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586329    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586331    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586333    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586336    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586340    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586342    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586345    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586347    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586350    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586353    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586356    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586358    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586361    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586363    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586366    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:29:23.588649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586368    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586371    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586374    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586376    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586379    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586381    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586384    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586386    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586389    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586392    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586395    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586398    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586401    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586403    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586406    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586408    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586411    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586413    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586416    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586419    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:29:23.589154 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586421    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586424    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586426    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586430    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586432    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586434    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586438    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586441    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586443    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586446    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586448    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586451    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586453    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586456    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586458    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586462    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586466    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586469    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586471    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586474    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:29:23.589662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586478    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586482    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586485    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586834    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586839    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586841    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586844    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586847    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586850    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586852    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586855    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586858    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586860    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586864    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586866    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586869    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586871    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586874    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586878    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586880    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:29:23.590177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586883    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586886    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586888    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586891    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586893    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586898    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586902    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586905    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586907    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586910    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586913    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586916    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586919    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586922    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586925    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586928    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586931    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586934    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586938    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:29:23.590668 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586942    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586946    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586948    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586951    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586954    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586956    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586959    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586961    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586964    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586966    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586969    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586972    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586975    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586977    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586980    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586982    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586985    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586987    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586990    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586992    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:29:23.591134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586995    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.586998    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587000    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587003    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587005    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587008    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587010    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587014    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587016    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587019    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587022    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587024    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587027    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587029    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587031    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587034    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587036    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587039    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587041    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587044    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:29:23.591649 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587046    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587049    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587051    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587054    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587057    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587059    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587062    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587064    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587067    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587069    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587073    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587077    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587080    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587083    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587085    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587088    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587091    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587094    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587097    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:29:23.592140 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587114    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587117    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587120    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587124    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587190    2571 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587227    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587233    2571 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587237    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587241    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587244    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587248    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587252    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587256    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587259    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587263    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587266    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587270    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587273    2571 flags.go:64] FLAG: --cgroup-root=""
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587276    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587279    2571 flags.go:64] FLAG: --client-ca-file=""
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587282    2571 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587284    2571 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587288    2571 flags.go:64] FLAG: --cluster-domain=""
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587293    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 19:29:23.592606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587296    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587300    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587303    2571 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587307    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587310    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587313    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587316    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587319    2571 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587321    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587324    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587327    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587330    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587335    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587338    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587341    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587345    2571 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587348    2571 flags.go:64] FLAG: --enable-server="true"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587351    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587355    2571 flags.go:64] FLAG: --event-burst="100"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587358    2571 flags.go:64] FLAG: --event-qps="50"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587361    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587364    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587367    2571 flags.go:64] FLAG: --eviction-hard=""
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587371    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 19:29:23.593258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587374    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587377    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587379    2571 flags.go:64] FLAG: --eviction-soft=""
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587382    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587385    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587388    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587391    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587395    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587398    2571 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587401    2571 flags.go:64] FLAG: --feature-gates=""
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587404    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587407    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587410    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587413    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587416    2571 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587419    2571 flags.go:64] FLAG: --help="false"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587422    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-136-27.ec2.internal"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587425    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587428    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587431    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587434    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587438    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587441    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 19:29:23.593853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587444    2571 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587447    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587450    2571 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587453    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587456    2571 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587459    2571 flags.go:64] FLAG: --kube-reserved=""
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587462    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587464    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587467    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587470    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587473    2571 flags.go:64] FLAG: --lock-file=""
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587476    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587478    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587481    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587486    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587489    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587494    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587496    2571 flags.go:64] FLAG: --logging-format="text"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587499    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587503    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587505    2571 flags.go:64] FLAG: --manifest-url=""
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587508    2571 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587512    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587515    2571 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587519    2571 flags.go:64] FLAG: --max-pods="110"
Sep 11 19:29:23.594729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587522    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587525    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587528    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587531    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587534    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587537    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587540    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587548    2571 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587551    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587554    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587557    2571 flags.go:64] FLAG: --pod-cidr=""
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587560    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587565    2571 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587567    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587570    2571 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587573    2571 flags.go:64] FLAG: --port="10250"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587576    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587579    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04e79ba27c0a4fccf"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587583    2571 flags.go:64] FLAG: --qos-reserved=""
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587586    2571 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587588    2571 flags.go:64] FLAG: --register-node="true"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587591    2571 flags.go:64] FLAG: --register-with-taints=""
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587595    2571 flags.go:64] FLAG: --registry-burst="10"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587598    2571 flags.go:64] FLAG: --registry-qps="5"
Sep 11 19:29:23.595644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587602    2571 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587605    2571 flags.go:64] FLAG: --reserved-memory=""
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587609    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587612    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587615    2571 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587618    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587620    2571 flags.go:64] FLAG: --runonce="false"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587623    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587626    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587629    2571 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587632    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587634    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587637    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587640    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587643    2571 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587648    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587651    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587654    2571 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587656    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587660    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587663    2571 flags.go:64] FLAG: --system-cgroups=""
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587666    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587671    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587673    2571 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587676    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 19:29:23.596275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587680    2571 flags.go:64] FLAG: --tls-min-version=""
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587683    2571 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587686    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587689    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587692    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587694    2571 flags.go:64] FLAG: --v="2"
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587698    2571 flags.go:64] FLAG: --version="false"
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587703    2571 flags.go:64] FLAG: --vmodule=""
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587707    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.587710    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587824    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587828    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587831    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587835    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587839    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587842    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587845    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587847    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587850    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587853    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587857    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:29:23.596890 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587860    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587865    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587868    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587871    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587874    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587876    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587879    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587882    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587884    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587887    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587889    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587892    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587894    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587897    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587900    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587902    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587905    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587907    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587911    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587913    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:29:23.597453 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587916    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587919    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587922    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587924    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587927    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587930    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587932    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587935    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587939    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587942    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587945    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587948    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587951    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587955    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587958    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587961    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587963    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587966    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587968    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:29:23.597956 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587971    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587973    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587977    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587979    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587982    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587984    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587987    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587990    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587992    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587995    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.587997    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588001    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588004    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588006    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588009    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588012    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588014    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588017    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588019    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:29:23.598474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588022    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588025    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588027    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588030    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588032    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588035    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588038    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588041    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588044    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588046    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588049    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588052    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588055    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588057    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588060    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588062    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588065    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588067    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588070    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588072    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:29:23.598938 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588075    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588078    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588080    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588083    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588087    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588089    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588092    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588094    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588097    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.588112    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.588118    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.597140    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.597156    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.597816    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.597828    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 19:29:23.599467 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597884    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597889    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597892    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597895    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597898    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597902    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597905    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597907    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597910    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597913    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597917    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597921    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597924    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597928    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597930    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597934    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597936    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597939    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597942    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597945    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:29:23.599844 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597949    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597951    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597954    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597957    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597960    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597963    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597966    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597968    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597971    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597975    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597978    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597981    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597983    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597986    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597990    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597993    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.597997    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598001    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598004    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:29:23.600362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598007    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598010    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598013    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598016    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598018    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598021    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598023    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598027    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598030    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598033    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598035    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598038    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598041    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598044    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598046    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598049    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598051    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598054    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598057    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598059    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:29:23.600835 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598062    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598065    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598068    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598071    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598073    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598076    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598079    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598081    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598084    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598087    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598089    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598092    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598094    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598097    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598113    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598116    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598118    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598121    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598124    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598127    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:29:23.601341 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598130    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598133    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598135    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598138    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598140    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598143    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598146    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598148    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598151    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598153    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598156    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598159    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598161    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598164    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598168    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598170    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598173    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598176    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598178    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:29:23.601853 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598181    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.598186    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598296    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598301    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598304    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598307    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598310    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598313    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598316    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598319    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598321    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598324    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598327    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598329    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598332    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:29:23.602362 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598335    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598338    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598340    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598342    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598345    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598348    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598350    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598353    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598356    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598359    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598361    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598364    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598367    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598369    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598372    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598375    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598377    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598380    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598382    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598385    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:29:23.602735 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598388    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598390    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598393    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598395    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598398    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598400    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598403    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598405    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598408    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598410    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598413    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598416    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598419    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598422    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598424    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598427    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598429    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598432    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598434    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598437    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:29:23.603249 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598439    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598442    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598445    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598447    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598450    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598453    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598456    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598458    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598461    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598464    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598466    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598469    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598471    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598474    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598476    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598479    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598482    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598484    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598487    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:29:23.603740 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598491    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598495    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598498    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598501    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598504    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598507    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598510    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598513    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598515    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598518    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598521    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598523    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598526    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598528    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598531    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598534    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598536    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598539    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598543    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:29:23.604259 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598547    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598551    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598554    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598557    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598560    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598563    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598565    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:23.598568    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.598573    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.598673    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.602071    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.603030    2571 server.go:1019] "Starting client certificate rotation"
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.603126    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 19:29:23.604717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.603160    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 19:29:23.605449 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.605436    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 19:29:23.610198 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.610154    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 19:29:23.619742 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.619727    2571 log.go:25] "Validated CRI v1 runtime API"
Sep 11 19:29:23.623837 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.623822    2571 log.go:25] "Validated CRI v1 image API"
Sep 11 19:29:23.626586 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.626561    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 19:29:23.629733 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.629712    2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c7d800c6-da8d-4fae-994e-ee4ad7b0b889:/dev/nvme0n1p3 e90d71ff-ce2b-4e69-84ef-5173dc27c384:/dev/nvme0n1p4]
Sep 11 19:29:23.629777 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.629734    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 19:29:23.636644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.636269    2571 manager.go:216] Machine: {Timestamp:2026-09-11 19:29:23.634648813 +0000 UTC m=+0.310585710 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3231744 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dd9690989cc89547264af2dc5af0e SystemUUID:ec2dd969-0989-cc89-5472-64af2dc5af0e BootID:d93dbacc-ae19-4328-8fe4-83502766f73e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f4:b9:1c:12:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f4:b9:1c:12:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:6d:3e:ff:e6:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 19:29:23.636644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.636635    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 19:29:23.636766 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.636726    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 19:29:23.637486 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.637470    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 19:29:23.639378 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.639354    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 19:29:23.639502 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.639381    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 19:29:23.640488 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.640478    2571 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 19:29:23.640525 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.640490    2571 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 19:29:23.640525 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.640499    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 19:29:23.641178 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.641167    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 19:29:23.641998 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.641989    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 19:29:23.645739 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.645728    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 19:29:23.645846 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.645837    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 19:29:23.645875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.645865    2571 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 19:29:23.645875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.645873    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 19:29:23.645927 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.645887    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 19:29:23.645927 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.645896    2571 kubelet.go:398] "Adding apiserver pod source"
Sep 11 19:29:23.645927 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.645906    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 19:29:23.646237 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.646226    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 19:29:23.646276 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.646244    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 19:29:23.648403 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.648387    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 19:29:23.650345 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.650331    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 19:29:23.650413 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.650360    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 19:29:23.652275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652261    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 19:29:23.652275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652278    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652284    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652290    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652301    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652307    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652320    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652325    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652331    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652336    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652344    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 19:29:23.652370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.652350    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 19:29:23.653037 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.653014    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 19:29:23.653037 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.653028    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 19:29:23.655478 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.655458    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-27.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 19:29:23.655478 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.655459    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 19:29:23.655644 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.655602    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 19:29:23.656880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.656866    2571 server.go:1268] "Started kubelet"
Sep 11 19:29:23.656962 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.656938    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 19:29:23.657260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.657048    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 19:29:23.657409 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.657393    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 19:29:23.657905 ip-10-0-136-27 systemd[1]: Started Kubernetes Kubelet.
Sep 11 19:29:23.658537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.658508    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 19:29:23.659503 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.659488    2571 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 19:29:23.663207 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.663191    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 19:29:23.663655 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.663634    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 19:29:23.664388 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.664119    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:23.664461 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.664355    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 19:29:23.664461 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.664401    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 19:29:23.664589 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.664376    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 19:29:23.664656 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.664642    2571 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 19:29:23.664710 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.664657    2571 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 19:29:23.664710 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.664699    2571 factory.go:55] Registering systemd factory
Sep 11 19:29:23.664799 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.664732    2571 factory.go:223] Registration of the systemd container factory successfully
Sep 11 19:29:23.664949 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.663146    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-27.ec2.internal.18d45af5c8e4855b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-27.ec2.internal,UID:ip-10-0-136-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-27.ec2.internal,},FirstTimestamp:2026-09-11 19:29:23.656836443 +0000 UTC m=+0.332773343,LastTimestamp:2026-09-11 19:29:23.656836443 +0000 UTC m=+0.332773343,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-27.ec2.internal,}"
Sep 11 19:29:23.665202 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.665186    2571 factory.go:153] Registering CRI-O factory
Sep 11 19:29:23.665202 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.665204    2571 factory.go:223] Registration of the crio container factory successfully
Sep 11 19:29:23.665343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.665256    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 19:29:23.665343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.665279    2571 factory.go:103] Registering Raw factory
Sep 11 19:29:23.665343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.665292    2571 manager.go:1196] Started watching for new ooms in manager
Sep 11 19:29:23.665700 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.665682    2571 manager.go:349] Starting recovery of all containers
Sep 11 19:29:23.665790 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.665701    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 19:29:23.670848 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.670822    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 19:29:23.670970 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.670845    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 19:29:23.679064 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.679044    2571 manager.go:354] Recovery completed
Sep 11 19:29:23.684890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.684873    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:29:23.686511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.686493    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wh9pl"
Sep 11 19:29:23.686765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.686751    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:29:23.686806 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.686776    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:29:23.686806 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.686786    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:29:23.687316 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.687301    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 19:29:23.687316 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.687310    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:29:23.687316 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.687315    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 19:29:23.687465 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.687344    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 19:29:23.688028 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.688015    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:29:23.688078 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.688043    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:29:23.688078 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.688054    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:29:23.688865 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.688798    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-27.ec2.internal.18d45af5caad3184  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-27.ec2.internal,UID:ip-10-0-136-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-27.ec2.internal,},FirstTimestamp:2026-09-11 19:29:23.686764932 +0000 UTC m=+0.362701831,LastTimestamp:2026-09-11 19:29:23.686764932 +0000 UTC m=+0.362701831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-27.ec2.internal,}"
Sep 11 19:29:23.695219 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.695200    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wh9pl"
Sep 11 19:29:23.696978 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.696908    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-27.ec2.internal.18d45af5caad6ea6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-27.ec2.internal,UID:ip-10-0-136-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-27.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-27.ec2.internal,},FirstTimestamp:2026-09-11 19:29:23.686780582 +0000 UTC m=+0.362717481,LastTimestamp:2026-09-11 19:29:23.686780582 +0000 UTC m=+0.362717481,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-27.ec2.internal,}"
Sep 11 19:29:23.699768 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.699700    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-27.ec2.internal.18d45af5caad920b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-27.ec2.internal,UID:ip-10-0-136-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-27.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-27.ec2.internal,},FirstTimestamp:2026-09-11 19:29:23.686789643 +0000 UTC m=+0.362726542,LastTimestamp:2026-09-11 19:29:23.686789643 +0000 UTC m=+0.362726542,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-27.ec2.internal,}"
Sep 11 19:29:23.704488 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.704415    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-27.ec2.internal.18d45af5caad3184\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-27.ec2.internal.18d45af5caad3184  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-27.ec2.internal,UID:ip-10-0-136-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-27.ec2.internal,},FirstTimestamp:2026-09-11 19:29:23.686764932 +0000 UTC m=+0.362701831,LastTimestamp:2026-09-11 19:29:23.688028446 +0000 UTC m=+0.363965347,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-27.ec2.internal,}"
Sep 11 19:29:23.714014 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.713939    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-27.ec2.internal.18d45af5caad6ea6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-27.ec2.internal.18d45af5caad6ea6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-27.ec2.internal,UID:ip-10-0-136-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-27.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-27.ec2.internal,},FirstTimestamp:2026-09-11 19:29:23.686780582 +0000 UTC m=+0.362717481,LastTimestamp:2026-09-11 19:29:23.688048967 +0000 UTC m=+0.363985865,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-27.ec2.internal,}"
Sep 11 19:29:23.834813 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.721546    2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-27.ec2.internal.18d45af5caad920b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-27.ec2.internal.18d45af5caad920b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-27.ec2.internal,UID:ip-10-0-136-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-27.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-27.ec2.internal,},FirstTimestamp:2026-09-11 19:29:23.686789643 +0000 UTC m=+0.362726542,LastTimestamp:2026-09-11 19:29:23.688057711 +0000 UTC m=+0.363994611,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-27.ec2.internal,}"
Sep 11 19:29:23.834813 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.764609    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:23.834813 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.833699    2571 policy_none.go:49] "None policy: Start"
Sep 11 19:29:23.834813 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.833715    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 19:29:23.834813 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.833725    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 19:29:23.835675 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.835664    2571 policy_none.go:47] "Start"
Sep 11 19:29:23.865313 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.865287    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:23.865595 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.865576    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 19:29:23.866744 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.866730    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 19:29:23.866804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.866771    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 19:29:23.866804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.866791    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 19:29:23.866905 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.866824    2571 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 11 19:29:23.870673 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.870659    2571 manager.go:337] "Starting Device Plugin manager"
Sep 11 19:29:23.870770 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.870699    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 19:29:23.870770 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.870712    2571 server.go:87] "Starting device plugin registration server"
Sep 11 19:29:23.871015 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.871000    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 19:29:23.871122 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.871015    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 19:29:23.871247 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.871228    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 19:29:23.871345 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.871327    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 19:29:23.871410 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.871347    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 19:29:23.871752 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.871730    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 19:29:23.871818 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.871788    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:23.880464 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.880448    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:29:23.884369 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.884352    2571 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-27.ec2.internal\" not found" node="ip-10-0-136-27.ec2.internal"
Sep 11 19:29:23.967769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.967738    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal"]
Sep 11 19:29:23.967844 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.967805    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:29:23.969111 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.969083    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:29:23.969186 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.969126    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:29:23.969186 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.969136    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:29:23.971202 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.971189    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:29:23.971339 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.971326    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:29:23.971411 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.971402    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:23.971449 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.971427    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:29:23.973678 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973664    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:29:23.973762 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973684    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:29:23.973762 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973705    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:29:23.973762 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973714    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:29:23.973762 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973688    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:29:23.973900 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973768    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:29:23.973900 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973779    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:29:23.973900 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973797    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-27.ec2.internal"
Sep 11 19:29:23.973900 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973691    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:29:23.973900 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.973836    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:29:23.975937 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.975920    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:23.976011 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.975950    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:29:23.976570 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.976555    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:29:23.976670 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.976583    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:29:23.976670 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.976596    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:29:23.983312 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:23.983297    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-27.ec2.internal"
Sep 11 19:29:23.983388 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.983315    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-27.ec2.internal\": node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:23.995634 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.995616    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-27.ec2.internal\" not found" node="ip-10-0-136-27.ec2.internal"
Sep 11 19:29:23.998850 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:23.998836    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-27.ec2.internal\" not found" node="ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.013500 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.013483    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.067577 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.067555    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b46452aba430ae639fdf33b411e2c0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal\" (UID: \"2b46452aba430ae639fdf33b411e2c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.067659 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.067578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b46452aba430ae639fdf33b411e2c0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal\" (UID: \"2b46452aba430ae639fdf33b411e2c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.067659 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.067596    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496e13ce30a424109c88ee9e45a70e85-config\") pod \"kube-apiserver-proxy-ip-10-0-136-27.ec2.internal\" (UID: \"496e13ce30a424109c88ee9e45a70e85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.113978 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.113931    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.168496 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.168447    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b46452aba430ae639fdf33b411e2c0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal\" (UID: \"2b46452aba430ae639fdf33b411e2c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.168496 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.168500    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b46452aba430ae639fdf33b411e2c0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal\" (UID: \"2b46452aba430ae639fdf33b411e2c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.168686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.168541    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b46452aba430ae639fdf33b411e2c0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal\" (UID: \"2b46452aba430ae639fdf33b411e2c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.168686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.168545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496e13ce30a424109c88ee9e45a70e85-config\") pod \"kube-apiserver-proxy-ip-10-0-136-27.ec2.internal\" (UID: \"496e13ce30a424109c88ee9e45a70e85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.168686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.168587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/496e13ce30a424109c88ee9e45a70e85-config\") pod \"kube-apiserver-proxy-ip-10-0-136-27.ec2.internal\" (UID: \"496e13ce30a424109c88ee9e45a70e85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.168686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.168605    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b46452aba430ae639fdf33b411e2c0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal\" (UID: \"2b46452aba430ae639fdf33b411e2c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.214565 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.214538    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.297972 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.297950    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.301190 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.301172    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.314644 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.314627    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.418291 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.418266    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.522032 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.522006    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.604176 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.604152    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 19:29:24.604563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.604298    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 19:29:24.622233 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.622208    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.723361 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.723301    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.763493 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.763463    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:29:24.824555 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:24.824529    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-27.ec2.internal\" not found"
Sep 11 19:29:24.834465 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.834433    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 19:24:24 +0000 UTC" deadline="2028-06-20 04:58:21.322204906 +0000 UTC"
Sep 11 19:29:24.834553 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.834482    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15537h28m56.487743989s"
Sep 11 19:29:24.866071 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.866052    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:29:24.966571 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.966550    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.981003 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.980956    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 19:29:24.982658 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.982644    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal"
Sep 11 19:29:24.992452 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:24.992432    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 19:29:25.013883 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.013865    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:29:25.166138 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:25.166096    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b46452aba430ae639fdf33b411e2c0a.slice/crio-012d7cef66497b300031661de6300f5fa77b15d9c6fb55af901cc6de691b586e WatchSource:0}: Error finding container 012d7cef66497b300031661de6300f5fa77b15d9c6fb55af901cc6de691b586e: Status 404 returned error can't find the container with id 012d7cef66497b300031661de6300f5fa77b15d9c6fb55af901cc6de691b586e
Sep 11 19:29:25.174248 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.174230    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:29:25.179439 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:25.179414    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496e13ce30a424109c88ee9e45a70e85.slice/crio-0e57e050f0496c4cea4e6ed3cf072d235dfdbdfad1d56fd43e79413472d9e377 WatchSource:0}: Error finding container 0e57e050f0496c4cea4e6ed3cf072d235dfdbdfad1d56fd43e79413472d9e377: Status 404 returned error can't find the container with id 0e57e050f0496c4cea4e6ed3cf072d235dfdbdfad1d56fd43e79413472d9e377
Sep 11 19:29:25.647910 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.647876    2571 apiserver.go:52] "Watching apiserver"
Sep 11 19:29:25.655784 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.655757    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 19:29:25.656165 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.656142    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal","openshift-multus/multus-6s5tc","openshift-multus/network-metrics-daemon-mtpxz","kube-system/konnectivity-agent-n22pk","openshift-cluster-node-tuning-operator/tuned-rg5hg","openshift-multus/multus-additional-cni-plugins-mk9z2","openshift-network-diagnostics/network-check-target-gs762","openshift-network-operator/iptables-alerter-z6v8q","openshift-ovn-kubernetes/ovnkube-node-pj8nm","kube-system/global-pull-secret-syncer-z292n","kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln","openshift-dns/node-resolver-wmckq","openshift-image-registry/node-ca-xg2gb"]
Sep 11 19:29:25.659019 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.659001    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:25.659112 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.659092    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:25.661199 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.661180    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.663563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.663540    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:25.663671 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.663599    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 19:29:25.663671 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.663604    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:25.664055 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.664039    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 19:29:25.664154 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.664058    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2k7mj\""
Sep 11 19:29:25.664154 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.664046    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 19:29:25.665725 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.665708    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:25.668099 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.668031    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.668660 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.668640    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 19:29:25.668738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.668642    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 19:29:25.668738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.668694    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rk6ll\""
Sep 11 19:29:25.670582 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.670561    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.670851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.670828    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jz2v5\""
Sep 11 19:29:25.671020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.671002    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:29:25.671079 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.671003    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 19:29:25.672947 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.672930    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.673645 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.673626    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 19:29:25.673738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.673628    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 19:29:25.673738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.673652    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 19:29:25.673738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.673632    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 19:29:25.673738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.673734    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vnpcr\""
Sep 11 19:29:25.675021 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.675006    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 19:29:25.675517 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.675500    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.676411 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25cf9814-dfe6-41da-b125-56e077f8faf8-tmp-dir\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.676520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4nw\" (UniqueName: \"kubernetes.io/projected/25cf9814-dfe6-41da-b125-56e077f8faf8-kube-api-access-5s4nw\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.676520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d24c309d-5363-43b8-a580-5e9c3c11de02-agent-certs\") pod \"konnectivity-agent-n22pk\" (UID: \"d24c309d-5363-43b8-a580-5e9c3c11de02\") " pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:25.676520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:25.676680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkw65\" (UniqueName: \"kubernetes.io/projected/d81d2d85-34c6-44f9-a0e4-17ae596fea59-kube-api-access-lkw65\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:25.676680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d24c309d-5363-43b8-a580-5e9c3c11de02-konnectivity-ca\") pod \"konnectivity-agent-n22pk\" (UID: \"d24c309d-5363-43b8-a580-5e9c3c11de02\") " pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:25.676680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676597    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 19:29:25.676680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676605    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:25.676680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.676631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25cf9814-dfe6-41da-b125-56e077f8faf8-hosts-file\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.677853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.677533    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 19:29:25.677853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.677641    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 19:29:25.677853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.677712    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c6fd7\""
Sep 11 19:29:25.678038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.678002    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 19:29:25.678779 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.678756    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqhgm\""
Sep 11 19:29:25.679077 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.679053    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 19:29:25.679158 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.678764    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 19:29:25.679158 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.679129    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 19:29:25.679287 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.679164    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 19:29:25.679568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.679546    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.679628 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.679608    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 19:29:25.679678 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.679653    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 19:29:25.681546 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.681528    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.681656 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.681638    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:25.682520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.682501    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 19:29:25.682920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.682906    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dvpst\""
Sep 11 19:29:25.683972 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.683954    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.686403 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.686387    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.686791 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.686766    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 19:29:25.686791 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.686785    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rjgk5\""
Sep 11 19:29:25.687001 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.686829    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 19:29:25.687001 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.686858    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:29:25.689216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.689194    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 19:29:25.689216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.689213    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 19:29:25.689365 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.689217    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xcv6q\""
Sep 11 19:29:25.689480 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.689465    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 19:29:25.704583 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.704567    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7gzqd"
Sep 11 19:29:25.713029 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.713014    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7gzqd"
Sep 11 19:29:25.765808 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.765786    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 19:29:25.777145 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777121    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovnkube-config\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.777236 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777148    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-kubelet\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.777236 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777164    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-etc-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.777236 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777193    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysctl-d\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.777357 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777232    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-tuned\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.777357 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-host\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.777357 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777294    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5rx\" (UniqueName: \"kubernetes.io/projected/6be83e60-4480-4499-9053-044eb1a506f9-kube-api-access-xj5rx\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.777357 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777324    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d24c309d-5363-43b8-a580-5e9c3c11de02-konnectivity-ca\") pod \"konnectivity-agent-n22pk\" (UID: \"d24c309d-5363-43b8-a580-5e9c3c11de02\") " pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:25.777357 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777343    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25cf9814-dfe6-41da-b125-56e077f8faf8-tmp-dir\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.777568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/412e8257-acb6-41f7-80de-b26261cb8fcb-dbus\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.777568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777410    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.777568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777428    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psstp\" (UniqueName: \"kubernetes.io/projected/78431f60-5d8c-477d-ab18-5e640b560f2b-kube-api-access-psstp\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.777568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777479    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.777568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777536    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-sys-fs\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.777568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777562    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-sys\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.777773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777588    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-slash\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.777773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25cf9814-dfe6-41da-b125-56e077f8faf8-tmp-dir\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.777773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-run-netns\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.777773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.777773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777698    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-host\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.777773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.777971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-system-cni-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.777971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-k8s-cni-cncf-io\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.777971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-cni-multus\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.777971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777869    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4nw\" (UniqueName: \"kubernetes.io/projected/25cf9814-dfe6-41da-b125-56e077f8faf8-kube-api-access-5s4nw\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.777971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777907    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovn-node-metrics-cert\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.777971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777933    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovnkube-script-lib\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.777971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777954    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d24c309d-5363-43b8-a580-5e9c3c11de02-konnectivity-ca\") pod \"konnectivity-agent-n22pk\" (UID: \"d24c309d-5363-43b8-a580-5e9c3c11de02\") " pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:25.777971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777966    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.777993    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be83e60-4480-4499-9053-044eb1a506f9-multus-daemon-config\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778020    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-conf-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778054    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-var-lib-kubelet\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-systemd\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778111    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-var-lib-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778134    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78431f60-5d8c-477d-ab18-5e640b560f2b-host-slash\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-registration-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778226    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-modprobe-d\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.778266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778249    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtsh\" (UniqueName: \"kubernetes.io/projected/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-kube-api-access-kvtsh\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778309    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-systemd-units\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-cni-netd\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-cnibin\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-cni-binary-copy\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778447    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-netns\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-hostroot\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.778491    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:25.778578 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.778572    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:26.27855008 +0000 UTC m=+2.954486990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778610    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-log-socket\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/412e8257-acb6-41f7-80de-b26261cb8fcb-kubelet-config\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778682    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-socket-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qg8n\" (UniqueName: \"kubernetes.io/projected/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-kube-api-access-9qg8n\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-cni-bin\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-env-overrides\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778805    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysctl-conf\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778824    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-socket-dir-parent\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-kubelet\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-device-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778913    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-multus-certs\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.778945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-ovn\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.778959    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-kubernetes\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779030    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-tmp\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-etc-kubernetes\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779114    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-cni-bin\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7kj\" (UniqueName: \"kubernetes.io/projected/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-kube-api-access-pc7kj\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779191    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2hp\" (UniqueName: \"kubernetes.io/projected/17ed7e55-0604-462b-99f8-eb3e017873e0-kube-api-access-2t2hp\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysconfig\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-run\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779268    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779310    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-systemd\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9646m\" (UniqueName: \"kubernetes.io/projected/fb06fd55-d2a4-46c2-89e8-9247da15014f-kube-api-access-9646m\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.779373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be83e60-4480-4499-9053-044eb1a506f9-cni-binary-copy\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.779876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25cf9814-dfe6-41da-b125-56e077f8faf8-hosts-file\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.779876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779408    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-lib-modules\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.779876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779425    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-system-cni-dir\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.779876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779427    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25cf9814-dfe6-41da-b125-56e077f8faf8-hosts-file\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.779876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779464    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d24c309d-5363-43b8-a580-5e9c3c11de02-agent-certs\") pod \"konnectivity-agent-n22pk\" (UID: \"d24c309d-5363-43b8-a580-5e9c3c11de02\") " pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:25.779876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkw65\" (UniqueName: \"kubernetes.io/projected/d81d2d85-34c6-44f9-a0e4-17ae596fea59-kube-api-access-lkw65\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:25.780211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-node-log\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.780211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.779990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78431f60-5d8c-477d-ab18-5e640b560f2b-iptables-alerter-script\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.780211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.780081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.780211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.780124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-serviceca\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.780211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.780184    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-cni-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.780459 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.780217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-cnibin\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.780741 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.780706    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 19:29:25.785131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.785007    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d24c309d-5363-43b8-a580-5e9c3c11de02-agent-certs\") pod \"konnectivity-agent-n22pk\" (UID: \"d24c309d-5363-43b8-a580-5e9c3c11de02\") " pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:25.785601 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.785585    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:29:25.785654 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.785603    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:29:25.785654 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.785613    2571 projected.go:196] Error preparing data for projected volume kube-api-access-k6gdc for pod openshift-network-diagnostics/network-check-target-gs762: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:25.785765 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.785658    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc podName:a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d nodeName:}" failed. No retries permitted until 2026-09-11 19:29:26.285645582 +0000 UTC m=+2.961582472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k6gdc" (UniqueName: "kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc") pod "network-check-target-gs762" (UID: "a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:25.786308 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.786291    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4nw\" (UniqueName: \"kubernetes.io/projected/25cf9814-dfe6-41da-b125-56e077f8faf8-kube-api-access-5s4nw\") pod \"node-resolver-wmckq\" (UID: \"25cf9814-dfe6-41da-b125-56e077f8faf8\") " pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.791721 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.791704    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkw65\" (UniqueName: \"kubernetes.io/projected/d81d2d85-34c6-44f9-a0e4-17ae596fea59-kube-api-access-lkw65\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:25.872608 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.872557    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal" event={"ID":"496e13ce30a424109c88ee9e45a70e85","Type":"ContainerStarted","Data":"0e57e050f0496c4cea4e6ed3cf072d235dfdbdfad1d56fd43e79413472d9e377"}
Sep 11 19:29:25.874534 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.874508    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal" event={"ID":"2b46452aba430ae639fdf33b411e2c0a","Type":"ContainerStarted","Data":"012d7cef66497b300031661de6300f5fa77b15d9c6fb55af901cc6de691b586e"}
Sep 11 19:29:25.880586 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.880553    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78431f60-5d8c-477d-ab18-5e640b560f2b-iptables-alerter-script\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.880707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.880587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.880707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.880619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-serviceca\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.880707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.880642    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-cni-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.880707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.880668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-cnibin\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.880923 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.880735    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-cnibin\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.881084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881061    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovnkube-config\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.881195 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881122    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-kubelet\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.881195 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881154    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-etc-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.881195 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881179    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysctl-d\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-tuned\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881228    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-host\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881229    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-serviceca\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881253    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5rx\" (UniqueName: \"kubernetes.io/projected/6be83e60-4480-4499-9053-044eb1a506f9-kube-api-access-xj5rx\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78431f60-5d8c-477d-ab18-5e640b560f2b-iptables-alerter-script\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881298    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881067    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-cni-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881305    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-etc-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.881362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/412e8257-acb6-41f7-80de-b26261cb8fcb-dbus\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psstp\" (UniqueName: \"kubernetes.io/projected/78431f60-5d8c-477d-ab18-5e640b560f2b-kube-api-access-psstp\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881411    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-kubelet\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-sys-fs\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881511    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-sys\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881514    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/412e8257-acb6-41f7-80de-b26261cb8fcb-dbus\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881456    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-host\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-slash\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881554    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881562    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysctl-d\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881583    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-sys-fs\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-run-netns\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881625    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-sys\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881626    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-slash\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881660    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovnkube-config\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.881810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881674    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881688    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-run-netns\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-host\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.881728    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:25.881778    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret podName:412e8257-acb6-41f7-80de-b26261cb8fcb nodeName:}" failed. No retries permitted until 2026-09-11 19:29:26.381760139 +0000 UTC m=+3.057697027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret") pod "global-pull-secret-syncer-z292n" (UID: "412e8257-acb6-41f7-80de-b26261cb8fcb") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-system-cni-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881824    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-k8s-cni-cncf-io\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-host\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-cni-multus\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881876    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovn-node-metrics-cert\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovnkube-script-lib\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881952    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be83e60-4480-4499-9053-044eb1a506f9-multus-daemon-config\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-cni-multus\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881976    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-conf-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.882768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882006    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-k8s-cni-cncf-io\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882013    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-var-lib-kubelet\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882072    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-var-lib-kubelet\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.881926    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-system-cni-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882246    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-systemd\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882343    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-var-lib-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882345    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-conf-dir\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882369    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78431f60-5d8c-477d-ab18-5e640b560f2b-host-slash\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882396    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-registration-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882403    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-systemd\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882420    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-modprobe-d\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882457    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78431f60-5d8c-477d-ab18-5e640b560f2b-host-slash\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882471    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtsh\" (UniqueName: \"kubernetes.io/projected/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-kube-api-access-kvtsh\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882504    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-var-lib-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-systemd-units\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.883612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-cni-netd\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-cnibin\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882603    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-modprobe-d\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882617    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-cni-binary-copy\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882643    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-netns\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882645    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be83e60-4480-4499-9053-044eb1a506f9-multus-daemon-config\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-registration-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovnkube-script-lib\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882699    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-cni-netd\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882704    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-netns\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882725    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-hostroot\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882744    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-cnibin\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-hostroot\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882812    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-log-socket\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882841    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/412e8257-acb6-41f7-80de-b26261cb8fcb-kubelet-config\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-socket-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.882988    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-log-socket\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883032    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/412e8257-acb6-41f7-80de-b26261cb8fcb-kubelet-config\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:25.884527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883035    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qg8n\" (UniqueName: \"kubernetes.io/projected/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-kube-api-access-9qg8n\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883072    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-cni-bin\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883100    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-env-overrides\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883154    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-socket-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883164    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysctl-conf\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-socket-dir-parent\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883197    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-systemd-units\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-kubelet\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883254    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-var-lib-cni-bin\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883265    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-device-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883293    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb06fd55-d2a4-46c2-89e8-9247da15014f-cni-binary-copy\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-multus-certs\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-ovn\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883350    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17ed7e55-0604-462b-99f8-eb3e017873e0-device-dir\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-multus-socket-dir-parent\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883390    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-host-run-multus-certs\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883433    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-kubernetes\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883451    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-kubelet\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.885313 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883468    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-ovn\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883390    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-kubernetes\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883474    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysctl-conf\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-tmp\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883535    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-etc-kubernetes\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883561    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-cni-bin\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7kj\" (UniqueName: \"kubernetes.io/projected/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-kube-api-access-pc7kj\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2hp\" (UniqueName: \"kubernetes.io/projected/17ed7e55-0604-462b-99f8-eb3e017873e0-kube-api-access-2t2hp\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883655    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-env-overrides\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883659    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysconfig\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883699    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-run\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-sysconfig\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883753    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-run\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-systemd\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883831    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9646m\" (UniqueName: \"kubernetes.io/projected/fb06fd55-d2a4-46c2-89e8-9247da15014f-kube-api-access-9646m\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883859    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be83e60-4480-4499-9053-044eb1a506f9-cni-binary-copy\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-lib-modules\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883911    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-system-cni-dir\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883970    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-node-log\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.883986    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-run-openvswitch\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884027    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be83e60-4480-4499-9053-044eb1a506f9-etc-kubernetes\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-cni-bin\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884088    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884118    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-lib-modules\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884124    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-node-log\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-system-cni-dir\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884216    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-systemd\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884289    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb06fd55-d2a4-46c2-89e8-9247da15014f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be83e60-4480-4499-9053-044eb1a506f9-cni-binary-copy\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-ovn-node-metrics-cert\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.884605    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-etc-tuned\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.886968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.886210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-tmp\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.890253 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.890228    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psstp\" (UniqueName: \"kubernetes.io/projected/78431f60-5d8c-477d-ab18-5e640b560f2b-kube-api-access-psstp\") pod \"iptables-alerter-z6v8q\" (UID: \"78431f60-5d8c-477d-ab18-5e640b560f2b\") " pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:25.890602 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.890578    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5rx\" (UniqueName: \"kubernetes.io/projected/6be83e60-4480-4499-9053-044eb1a506f9-kube-api-access-xj5rx\") pod \"multus-6s5tc\" (UID: \"6be83e60-4480-4499-9053-044eb1a506f9\") " pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:25.891818 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.891787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtsh\" (UniqueName: \"kubernetes.io/projected/379e9ed9-30a2-4c57-8b37-0d595a4b36b8-kube-api-access-kvtsh\") pod \"tuned-rg5hg\" (UID: \"379e9ed9-30a2-4c57-8b37-0d595a4b36b8\") " pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.896408 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.896263    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2hp\" (UniqueName: \"kubernetes.io/projected/17ed7e55-0604-462b-99f8-eb3e017873e0-kube-api-access-2t2hp\") pod \"aws-ebs-csi-driver-node-5q2ln\" (UID: \"17ed7e55-0604-462b-99f8-eb3e017873e0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:25.896816 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.896782    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qg8n\" (UniqueName: \"kubernetes.io/projected/c6b1a476-e4f1-4aa3-9252-3cffe7e7e903-kube-api-access-9qg8n\") pod \"node-ca-xg2gb\" (UID: \"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903\") " pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:25.897285 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.897263    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7kj\" (UniqueName: \"kubernetes.io/projected/5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f-kube-api-access-pc7kj\") pod \"ovnkube-node-pj8nm\" (UID: \"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:25.898241 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.897800    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9646m\" (UniqueName: \"kubernetes.io/projected/fb06fd55-d2a4-46c2-89e8-9247da15014f-kube-api-access-9646m\") pod \"multus-additional-cni-plugins-mk9z2\" (UID: \"fb06fd55-d2a4-46c2-89e8-9247da15014f\") " pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.972038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.972008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wmckq"
Sep 11 19:29:25.978274 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.978250    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:25.985614 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:25.985570    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cf9814_dfe6_41da_b125_56e077f8faf8.slice/crio-0ba0bb393babdc6ad976846751b085e97ff806dbfaebaf5bd13927190690a61c WatchSource:0}: Error finding container 0ba0bb393babdc6ad976846751b085e97ff806dbfaebaf5bd13927190690a61c: Status 404 returned error can't find the container with id 0ba0bb393babdc6ad976846751b085e97ff806dbfaebaf5bd13927190690a61c
Sep 11 19:29:25.986983 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.986815    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rg5hg"
Sep 11 19:29:25.991521 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.991219    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mk9z2"
Sep 11 19:29:25.993134 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:25.993104    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24c309d_5363_43b8_a580_5e9c3c11de02.slice/crio-06e35705bf7937c8dab59b2020182f4180ff8c2c2f5037e28270ee5822a5d1a5 WatchSource:0}: Error finding container 06e35705bf7937c8dab59b2020182f4180ff8c2c2f5037e28270ee5822a5d1a5: Status 404 returned error can't find the container with id 06e35705bf7937c8dab59b2020182f4180ff8c2c2f5037e28270ee5822a5d1a5
Sep 11 19:29:25.997797 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:25.997705    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:26.002958 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.002856    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xg2gb"
Sep 11 19:29:26.015505 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.015304    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z6v8q"
Sep 11 19:29:26.017541 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.015837    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6s5tc"
Sep 11 19:29:26.019845 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.019623    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln"
Sep 11 19:29:26.032210 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:26.032162    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78431f60_5d8c_477d_ab18_5e640b560f2b.slice/crio-3455592d0558e8b00a9a0a40d25a5724da3e018d9d3ed82db7310d8122250a65 WatchSource:0}: Error finding container 3455592d0558e8b00a9a0a40d25a5724da3e018d9d3ed82db7310d8122250a65: Status 404 returned error can't find the container with id 3455592d0558e8b00a9a0a40d25a5724da3e018d9d3ed82db7310d8122250a65
Sep 11 19:29:26.041542 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:26.041260    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be83e60_4480_4499_9053_044eb1a506f9.slice/crio-9994c0361ed67ec6ac125cb38d7fd0310634bfb83753c24be6ecc1a3662dc29d WatchSource:0}: Error finding container 9994c0361ed67ec6ac125cb38d7fd0310634bfb83753c24be6ecc1a3662dc29d: Status 404 returned error can't find the container with id 9994c0361ed67ec6ac125cb38d7fd0310634bfb83753c24be6ecc1a3662dc29d
Sep 11 19:29:26.047193 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:26.047158    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ed7e55_0604_462b_99f8_eb3e017873e0.slice/crio-843f3ab13cdd8f6d96b3b711aa7a21b4b3706ac1d6a0255fec92ec77cc98745f WatchSource:0}: Error finding container 843f3ab13cdd8f6d96b3b711aa7a21b4b3706ac1d6a0255fec92ec77cc98745f: Status 404 returned error can't find the container with id 843f3ab13cdd8f6d96b3b711aa7a21b4b3706ac1d6a0255fec92ec77cc98745f
Sep 11 19:29:26.149370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.149293    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:29:26.287214 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.287175    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:26.287401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.287229    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:26.287401 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:26.287340    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:26.287401 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:26.287387    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:29:26.287572 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:26.287407    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:29:26.287572 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:26.287414    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:27.287390995 +0000 UTC m=+3.963327883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:26.287572 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:26.287421    2571 projected.go:196] Error preparing data for projected volume kube-api-access-k6gdc for pod openshift-network-diagnostics/network-check-target-gs762: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:26.287572 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:26.287475    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc podName:a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d nodeName:}" failed. No retries permitted until 2026-09-11 19:29:27.287457344 +0000 UTC m=+3.963394251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6gdc" (UniqueName: "kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc") pod "network-check-target-gs762" (UID: "a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:26.387977 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.387941    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:26.388140 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:26.388098    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:26.388250 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:26.388179    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret podName:412e8257-acb6-41f7-80de-b26261cb8fcb nodeName:}" failed. No retries permitted until 2026-09-11 19:29:27.388163725 +0000 UTC m=+4.064100613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret") pod "global-pull-secret-syncer-z292n" (UID: "412e8257-acb6-41f7-80de-b26261cb8fcb") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:26.716495 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.716416    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:24:26 +0000 UTC" deadline="2028-02-10 13:42:39.642303596 +0000 UTC"
Sep 11 19:29:26.716495 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.716490    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12402h13m12.925818304s"
Sep 11 19:29:26.878289 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.878238    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" event={"ID":"fb06fd55-d2a4-46c2-89e8-9247da15014f","Type":"ContainerStarted","Data":"8d21bf60c0dc9efe331b35654a00538dbed3600c1325c9d8bb62ca85016b1be2"}
Sep 11 19:29:26.879632 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.879589    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n22pk" event={"ID":"d24c309d-5363-43b8-a580-5e9c3c11de02","Type":"ContainerStarted","Data":"06e35705bf7937c8dab59b2020182f4180ff8c2c2f5037e28270ee5822a5d1a5"}
Sep 11 19:29:26.881228 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.881198    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wmckq" event={"ID":"25cf9814-dfe6-41da-b125-56e077f8faf8","Type":"ContainerStarted","Data":"0ba0bb393babdc6ad976846751b085e97ff806dbfaebaf5bd13927190690a61c"}
Sep 11 19:29:26.882629 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.882604    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln" event={"ID":"17ed7e55-0604-462b-99f8-eb3e017873e0","Type":"ContainerStarted","Data":"843f3ab13cdd8f6d96b3b711aa7a21b4b3706ac1d6a0255fec92ec77cc98745f"}
Sep 11 19:29:26.884097 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.884073    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z6v8q" event={"ID":"78431f60-5d8c-477d-ab18-5e640b560f2b","Type":"ContainerStarted","Data":"3455592d0558e8b00a9a0a40d25a5724da3e018d9d3ed82db7310d8122250a65"}
Sep 11 19:29:26.885419 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.885392    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xg2gb" event={"ID":"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903","Type":"ContainerStarted","Data":"070f7d1a3ac61f9b09f59e3360c70e3816df092ab380abdc79427efc9016bce8"}
Sep 11 19:29:26.886925 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.886889    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rg5hg" event={"ID":"379e9ed9-30a2-4c57-8b37-0d595a4b36b8","Type":"ContainerStarted","Data":"b02ef97af3c85d8316565d0e6967a752495f2d1f1727befcb843e7aef596f1b2"}
Sep 11 19:29:26.888439 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.888406    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6s5tc" event={"ID":"6be83e60-4480-4499-9053-044eb1a506f9","Type":"ContainerStarted","Data":"9994c0361ed67ec6ac125cb38d7fd0310634bfb83753c24be6ecc1a3662dc29d"}
Sep 11 19:29:26.889740 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:26.889714    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"1747d305c11f4a4e867440ee241f3a53a06364cfb0a26a1378d4ae5c6c3bf7ae"}
Sep 11 19:29:27.294855 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:27.294821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:27.295036 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:27.294874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:27.295036 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.294995    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:27.295036 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.295027    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:29:27.295257 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.295053    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:29:27.295257 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.295067    2571 projected.go:196] Error preparing data for projected volume kube-api-access-k6gdc for pod openshift-network-diagnostics/network-check-target-gs762: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:27.295257 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.295077    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:29.295053036 +0000 UTC m=+5.970989927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:27.295257 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.295124    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc podName:a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d nodeName:}" failed. No retries permitted until 2026-09-11 19:29:29.295104551 +0000 UTC m=+5.971041458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6gdc" (UniqueName: "kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc") pod "network-check-target-gs762" (UID: "a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:27.395254 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:27.395217    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:27.395440 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.395421    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:27.395509 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.395487    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret podName:412e8257-acb6-41f7-80de-b26261cb8fcb nodeName:}" failed. No retries permitted until 2026-09-11 19:29:29.395466186 +0000 UTC m=+6.071403077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret") pod "global-pull-secret-syncer-z292n" (UID: "412e8257-acb6-41f7-80de-b26261cb8fcb") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:27.717308 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:27.717261    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:24:26 +0000 UTC" deadline="2028-06-03 10:36:44.745900404 +0000 UTC"
Sep 11 19:29:27.717308 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:27.717306    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15135h7m17.028599211s"
Sep 11 19:29:27.870375 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:27.870342    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:27.870544 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.870474    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:27.870933 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:27.870909    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:27.871057 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.871021    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:27.871128 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:27.871096    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:27.871223 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:27.871172    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:28.925075 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:28.925010    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal" event={"ID":"496e13ce30a424109c88ee9e45a70e85","Type":"ContainerStarted","Data":"4c2bf5243bfae799ea7b25f97ad377be1e6ed5105600eabf2a8810abf00e08f3"}
Sep 11 19:29:29.312115 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.312072    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:29.312302 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.312148    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:29.312302 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.312287    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:29.312302 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.312300    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:29:29.312492 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.312315    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:29:29.312492 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.312328    2571 projected.go:196] Error preparing data for projected volume kube-api-access-k6gdc for pod openshift-network-diagnostics/network-check-target-gs762: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:29.312492 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.312353    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:33.312334067 +0000 UTC m=+9.988270972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:29.312492 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.312373    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc podName:a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d nodeName:}" failed. No retries permitted until 2026-09-11 19:29:33.312363028 +0000 UTC m=+9.988299916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6gdc" (UniqueName: "kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc") pod "network-check-target-gs762" (UID: "a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:29.413132 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.413091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:29.413343 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.413249    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:29.413343 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.413324    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret podName:412e8257-acb6-41f7-80de-b26261cb8fcb nodeName:}" failed. No retries permitted until 2026-09-11 19:29:33.413293555 +0000 UTC m=+10.089230458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret") pod "global-pull-secret-syncer-z292n" (UID: "412e8257-acb6-41f7-80de-b26261cb8fcb") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:29.870149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.870118    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:29.870348 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.870274    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:29.870718 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.870694    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:29.870830 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.870809    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:29.870896 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.870872    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:29.870952 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:29.870942    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:29.935988 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.935954    2571 generic.go:356] "Generic (PLEG): container finished" podID="2b46452aba430ae639fdf33b411e2c0a" containerID="14ba3881d5c66a293134cfdb5945bb8473d0b7774f60309344fa0fb259dddfe7" exitCode=0
Sep 11 19:29:29.936903 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.936872    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal" event={"ID":"2b46452aba430ae639fdf33b411e2c0a","Type":"ContainerDied","Data":"14ba3881d5c66a293134cfdb5945bb8473d0b7774f60309344fa0fb259dddfe7"}
Sep 11 19:29:29.954380 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:29.953999    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-27.ec2.internal" podStartSLOduration=4.953981099 podStartE2EDuration="4.953981099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:28.949437846 +0000 UTC m=+5.625374760" watchObservedRunningTime="2026-09-11 19:29:29.953981099 +0000 UTC m=+6.629918012"
Sep 11 19:29:31.867512 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:31.867479    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:31.868012 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:31.867605    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:31.868012 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:31.867619    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:31.868012 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:31.867718    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:31.868012 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:31.867759    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:31.868012 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:31.867811    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:33.347902 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:33.347559    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:33.347902 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:33.347613    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:33.347902 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.347760    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:29:33.347902 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.347782    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:29:33.347902 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.347795    2571 projected.go:196] Error preparing data for projected volume kube-api-access-k6gdc for pod openshift-network-diagnostics/network-check-target-gs762: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:33.347902 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.347800    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:33.347902 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.347858    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc podName:a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d nodeName:}" failed. No retries permitted until 2026-09-11 19:29:41.347839268 +0000 UTC m=+18.023776159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6gdc" (UniqueName: "kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc") pod "network-check-target-gs762" (UID: "a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:33.347902 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.347878    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:41.347868459 +0000 UTC m=+18.023805346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:33.448906 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:33.448668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:33.448906 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.448847    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:33.448906 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.448913    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret podName:412e8257-acb6-41f7-80de-b26261cb8fcb nodeName:}" failed. No retries permitted until 2026-09-11 19:29:41.448895059 +0000 UTC m=+18.124831950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret") pod "global-pull-secret-syncer-z292n" (UID: "412e8257-acb6-41f7-80de-b26261cb8fcb") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:33.868699 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:33.868647    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:33.868870 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.868759    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:33.869042 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:33.869003    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:33.869251 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.869157    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:33.869251 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:33.869177    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:33.869366 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:33.869254    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:35.868299 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:35.868194    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:35.868299 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:35.868244    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:35.868824 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:35.868357    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:35.868824 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:35.868512    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:35.868824 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:35.868551    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:35.868824 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:35.868720    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:37.867176 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:37.867147    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:37.867176 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:37.867176    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:37.867693 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:37.867147    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:37.867693 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:37.867271    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:37.867693 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:37.867373    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:37.867693 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:37.867456    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:39.867492 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:39.867456    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:39.867492 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:39.867495    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:39.868006 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:39.867456    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:39.868006 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:39.867591    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:39.868006 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:39.867704    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:39.868006 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:39.867901    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:41.405664 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:41.405606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:41.405664 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:41.405654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:41.406294 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.405784    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:41.406294 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.405866    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:57.405844786 +0000 UTC m=+34.081781674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:41.406294 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.405794    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:29:41.406294 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.405904    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:29:41.406294 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.405918    2571 projected.go:196] Error preparing data for projected volume kube-api-access-k6gdc for pod openshift-network-diagnostics/network-check-target-gs762: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:41.406294 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.405963    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc podName:a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d nodeName:}" failed. No retries permitted until 2026-09-11 19:29:57.405951434 +0000 UTC m=+34.081888328 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6gdc" (UniqueName: "kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc") pod "network-check-target-gs762" (UID: "a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:41.506293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:41.506261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:41.506491 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.506408    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:41.506491 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.506483    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret podName:412e8257-acb6-41f7-80de-b26261cb8fcb nodeName:}" failed. No retries permitted until 2026-09-11 19:29:57.506465219 +0000 UTC m=+34.182402108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret") pod "global-pull-secret-syncer-z292n" (UID: "412e8257-acb6-41f7-80de-b26261cb8fcb") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:41.867543 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:41.867465    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:41.867711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:41.867465    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:41.867711 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.867610    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:41.867711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:41.867465    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:41.867875 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.867697    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:41.867875 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:41.867783    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:43.868826 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:43.868607    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:43.869279 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:43.868704    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:43.869279 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:43.868914    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:43.869279 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:43.869022    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:43.869279 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:43.868729    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:43.869279 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:43.869136    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:44.967656 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.967431    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rg5hg" event={"ID":"379e9ed9-30a2-4c57-8b37-0d595a4b36b8","Type":"ContainerStarted","Data":"fd6339d51f2c86f5cf1cce9fcd2b6188c61d734e4af7309354619de9e7b985dc"}
Sep 11 19:29:44.969058 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.969036    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6s5tc" event={"ID":"6be83e60-4480-4499-9053-044eb1a506f9","Type":"ContainerStarted","Data":"9c8e1e69ce03ce68c0b16bdba2a4ab77355478dc5342566cc8eca14b157b9278"}
Sep 11 19:29:44.971301 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.971236    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"cbc8b6759bd563893fe464d91b5c25768e9237210ebb0f3658019df5368ffa5a"}
Sep 11 19:29:44.971368 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.971313    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"2294948f5eccd0a60187bac0028981b191e0e02277c5e4be08635c07f43161d1"}
Sep 11 19:29:44.971423 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.971367    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"29b2addfb3bd0666b4dc45c9b020cb498a02594cf3645dca458804583bf90ac4"}
Sep 11 19:29:44.972899 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.972877    2571 generic.go:356] "Generic (PLEG): container finished" podID="fb06fd55-d2a4-46c2-89e8-9247da15014f" containerID="57962374c415edf1eed83f0102862bf34b3e759544e444fe03cd8778afae182f" exitCode=0
Sep 11 19:29:44.972975 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.972946    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" event={"ID":"fb06fd55-d2a4-46c2-89e8-9247da15014f","Type":"ContainerDied","Data":"57962374c415edf1eed83f0102862bf34b3e759544e444fe03cd8778afae182f"}
Sep 11 19:29:44.974511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.974483    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n22pk" event={"ID":"d24c309d-5363-43b8-a580-5e9c3c11de02","Type":"ContainerStarted","Data":"ebb001d0832739dfcefc1db70168448844cc8553059b87b4b31bbd930470a32c"}
Sep 11 19:29:44.976019 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.975997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wmckq" event={"ID":"25cf9814-dfe6-41da-b125-56e077f8faf8","Type":"ContainerStarted","Data":"25cfdc2cb808e70c6766ee276ef69229c22b26aed16c72428fea1e9adb3da7d0"}
Sep 11 19:29:44.977801 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.977781    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal" event={"ID":"2b46452aba430ae639fdf33b411e2c0a","Type":"ContainerStarted","Data":"ce055eb04d8b273e23d94a6f6424df5d1f77ee14a182687424421daa0b41d916"}
Sep 11 19:29:44.979122 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.979084    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln" event={"ID":"17ed7e55-0604-462b-99f8-eb3e017873e0","Type":"ContainerStarted","Data":"513e91bb74621aa42bd4ac4bc1d956a17246767eade877a81044c05ff7fcf8b2"}
Sep 11 19:29:44.980414 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:44.980393    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xg2gb" event={"ID":"c6b1a476-e4f1-4aa3-9252-3cffe7e7e903","Type":"ContainerStarted","Data":"bc6a8a8a519efb216fb1f3341097b0286cf93af9e5999aea8c5aa3cf3fced731"}
Sep 11 19:29:45.000683 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.000646    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n22pk" podStartSLOduration=2.58866203 podStartE2EDuration="21.000634399s" totalImagesPullingTime="18.411972369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:45.000311646 +0000 UTC m=+21.676248581" watchObservedRunningTime="2026-09-11 19:29:45.000634399 +0000 UTC m=+21.676571307"
Sep 11 19:29:45.000773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.000712    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rg5hg" podStartSLOduration=2.599219942 podStartE2EDuration="21.000707336s" totalImagesPullingTime="18.401487394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:44.986193385 +0000 UTC m=+21.662130288" watchObservedRunningTime="2026-09-11 19:29:45.000707336 +0000 UTC m=+21.676644243"
Sep 11 19:29:45.021253 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.021217    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xg2gb" podStartSLOduration=2.635152427 podStartE2EDuration="21.021204794s" totalImagesPullingTime="18.386052367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:45.020957324 +0000 UTC m=+21.696894242" watchObservedRunningTime="2026-09-11 19:29:45.021204794 +0000 UTC m=+21.697141701"
Sep 11 19:29:45.110806 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.110771    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6s5tc" podStartSLOduration=2.505247862 podStartE2EDuration="21.110757205s" totalImagesPullingTime="18.605509343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:45.110579779 +0000 UTC m=+21.786516688" watchObservedRunningTime="2026-09-11 19:29:45.110757205 +0000 UTC m=+21.786694112"
Sep 11 19:29:45.125350 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.125310    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-27.ec2.internal" podStartSLOduration=20.125297908 podStartE2EDuration="20.125297908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:45.125279197 +0000 UTC m=+21.801216104" watchObservedRunningTime="2026-09-11 19:29:45.125297908 +0000 UTC m=+21.801234817"
Sep 11 19:29:45.142063 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.142029    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wmckq" podStartSLOduration=2.760039542 podStartE2EDuration="21.142017786s" totalImagesPullingTime="18.381978244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:45.141633372 +0000 UTC m=+21.817570280" watchObservedRunningTime="2026-09-11 19:29:45.142017786 +0000 UTC m=+21.817954693"
Sep 11 19:29:45.867382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.867354    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:45.867491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.867354    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:45.867491 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:45.867458    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:45.867491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.867471    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:45.867610 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:45.867538    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:45.867610 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:45.867599    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:45.983684 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.983655    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z6v8q" event={"ID":"78431f60-5d8c-477d-ab18-5e640b560f2b","Type":"ContainerStarted","Data":"18989d937533dd4915ec43585d8e77f6d767433e41e9d5db58196d4b0fc81f6b"}
Sep 11 19:29:45.987458 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.987433    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"c3915964e40461596a5292386a65e56ee4a8d5d18a02667db295c821e92fcac7"}
Sep 11 19:29:45.987578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.987514    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"d4eb1b06e224eb716045c10be3332ad7d7fcb10a82c2d4792539aabd298ecf44"}
Sep 11 19:29:45.987578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:45.987552    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"8568a961dc4a2bcd27b8ba34c864536e7a6b9fc3158f62081732c49492833208"}
Sep 11 19:29:46.083862 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:46.083839    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 19:29:46.885883 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:46.885762    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T19:29:46.083859705Z","UUID":"0daa9b9c-4ec4-4514-b318-c3ec028bb0dc","Handler":null,"Name":"","Endpoint":""}
Sep 11 19:29:46.887570 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:46.887540    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 19:29:46.887694 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:46.887578    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 19:29:46.991548 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:46.991518    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln" event={"ID":"17ed7e55-0604-462b-99f8-eb3e017873e0","Type":"ContainerStarted","Data":"3b86eeaa25a90e1d101f48a39d50676e53c3a534fc5695abc0181c9869c099fc"}
Sep 11 19:29:47.867258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:47.867224    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:47.867446 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:47.867234    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:47.867446 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:47.867318    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:47.867446 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:47.867421    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:47.867623 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:47.867457    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:47.867623 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:47.867560    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:47.996365 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:47.996333    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln" event={"ID":"17ed7e55-0604-462b-99f8-eb3e017873e0","Type":"ContainerStarted","Data":"7c94893109c0f11efbadd62f5f598902057d3bfa65bfe8334629e0adcfabb29a"}
Sep 11 19:29:48.000078 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:48.000049    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"69a28cf91ab51fbf45455a5f31a0cf7abec0c311bea11ac14babe52da43bf2b6"}
Sep 11 19:29:48.013976 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:48.013930    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z6v8q" podStartSLOduration=9.454883784 podStartE2EDuration="24.013916153s" totalImagesPullingTime="14.559032369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:45.998577235 +0000 UTC m=+22.674514147" watchObservedRunningTime="2026-09-11 19:29:48.013916153 +0000 UTC m=+24.689853061"
Sep 11 19:29:48.014185 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:48.014165    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q2ln" podStartSLOduration=3.5073978649999997 podStartE2EDuration="24.014158483s" totalImagesPullingTime="20.506760618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:48.012561898 +0000 UTC m=+24.688498806" watchObservedRunningTime="2026-09-11 19:29:48.014158483 +0000 UTC m=+24.690095426"
Sep 11 19:29:49.867041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:49.867010    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:49.867041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:49.867035    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:49.867536 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:49.867035    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:49.867536 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:49.867130    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:49.867536 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:49.867219    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:49.867536 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:49.867310    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:49.931427 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:49.931222    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:49.931892 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:49.931874    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:50.008728 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:50.008695    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" event={"ID":"5a25a671-c9e1-494e-a9f9-cf9b2b7aae0f","Type":"ContainerStarted","Data":"b180924a8db30c2962bdf2d2ae118f719087dcf16a792e7d3d7fb1647b69c0c0"}
Sep 11 19:29:50.008966 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:50.008921    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:50.009381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:50.009362    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n22pk"
Sep 11 19:29:50.034690 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:50.034652    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm" podStartSLOduration=7.361789837 podStartE2EDuration="26.034638358s" totalImagesPullingTime="18.672848521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:29:50.033966094 +0000 UTC m=+26.709903001" watchObservedRunningTime="2026-09-11 19:29:50.034638358 +0000 UTC m=+26.710575265"
Sep 11 19:29:51.013622 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.013583    2571 generic.go:356] "Generic (PLEG): container finished" podID="fb06fd55-d2a4-46c2-89e8-9247da15014f" containerID="5a4caa20c03b0c3ddbd548b1095967822da5f7e1d259d653d5fa469c1d56a375" exitCode=0
Sep 11 19:29:51.014076 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.013672    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" event={"ID":"fb06fd55-d2a4-46c2-89e8-9247da15014f","Type":"ContainerDied","Data":"5a4caa20c03b0c3ddbd548b1095967822da5f7e1d259d653d5fa469c1d56a375"}
Sep 11 19:29:51.014480 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.014456    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:51.014705 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.014679    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:51.014872 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.014847    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:51.032928 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.032907    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:51.033350 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.033335    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:29:51.609882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.609845    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z292n"]
Sep 11 19:29:51.610024 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.609977    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:51.610129 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:51.610088    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:51.612833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.612810    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mtpxz"]
Sep 11 19:29:51.612942 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.612924    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:51.613047 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:51.613027    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:51.613500 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.613479    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gs762"]
Sep 11 19:29:51.613593 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:51.613558    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:51.613657 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:51.613636    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:52.867202 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:52.867174    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:52.867803 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:52.867269    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:52.867803 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:52.867299    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:52.867803 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:52.867358    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:53.019172 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:53.019139    2571 generic.go:356] "Generic (PLEG): container finished" podID="fb06fd55-d2a4-46c2-89e8-9247da15014f" containerID="5700863b3b67c80d6c2e24e5a98823162c375b563cd3ae633da118c493580ed3" exitCode=0
Sep 11 19:29:53.019331 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:53.019220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" event={"ID":"fb06fd55-d2a4-46c2-89e8-9247da15014f","Type":"ContainerDied","Data":"5700863b3b67c80d6c2e24e5a98823162c375b563cd3ae633da118c493580ed3"}
Sep 11 19:29:53.867893 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:53.867865    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:53.868213 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:53.867953    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:54.023778 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:54.023751    2571 generic.go:356] "Generic (PLEG): container finished" podID="fb06fd55-d2a4-46c2-89e8-9247da15014f" containerID="2023c5bd8442ca1eb11b22f3f9cb26b97c42362a68b7dab54b07112ad5e74304" exitCode=0
Sep 11 19:29:54.023920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:54.023789    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" event={"ID":"fb06fd55-d2a4-46c2-89e8-9247da15014f","Type":"ContainerDied","Data":"2023c5bd8442ca1eb11b22f3f9cb26b97c42362a68b7dab54b07112ad5e74304"}
Sep 11 19:29:54.868186 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:54.867991    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:54.868587 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:54.868023    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:54.868587 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:54.868381    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:54.868587 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:54.868261    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:55.867950 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:55.867878    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:55.868080 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:55.868012    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:29:56.867901 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:56.867860    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:56.868351 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:56.867860    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:56.868351 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:56.867975    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z292n" podUID="412e8257-acb6-41f7-80de-b26261cb8fcb"
Sep 11 19:29:56.868351 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:56.868031    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gs762" podUID="a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d"
Sep 11 19:29:57.420224 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.420192    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:57.420224 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.420226    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:57.420417 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.420328    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:57.420417 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.420388    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:29.420373725 +0000 UTC m=+66.096310610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:29:57.420508 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.420335    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:29:57.420508 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.420449    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:29:57.420508 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.420459    2571 projected.go:196] Error preparing data for projected volume kube-api-access-k6gdc for pod openshift-network-diagnostics/network-check-target-gs762: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:57.420508 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.420500    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc podName:a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d nodeName:}" failed. No retries permitted until 2026-09-11 19:30:29.42048938 +0000 UTC m=+66.096426266 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6gdc" (UniqueName: "kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc") pod "network-check-target-gs762" (UID: "a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:29:57.521013 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.520980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:57.521205 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.521113    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:57.521205 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.521181    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret podName:412e8257-acb6-41f7-80de-b26261cb8fcb nodeName:}" failed. No retries permitted until 2026-09-11 19:30:29.52116221 +0000 UTC m=+66.197099100 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret") pod "global-pull-secret-syncer-z292n" (UID: "412e8257-acb6-41f7-80de-b26261cb8fcb") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:29:57.653044 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.653019    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-27.ec2.internal" event="NodeReady"
Sep 11 19:29:57.653185 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.653151    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 19:29:57.687637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.687563    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bf7847486-ccpsb"]
Sep 11 19:29:57.729824 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.728964    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"]
Sep 11 19:29:57.729824 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.729191    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.732031 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.732007    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 19:29:57.732310 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.732285    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 19:29:57.732431 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.732332    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 19:29:57.732504 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.732485    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5wflx\""
Sep 11 19:29:57.738393 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.737991    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 19:29:57.743664 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.743647    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"]
Sep 11 19:29:57.743777 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.743760    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.746478 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.746456    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 19:29:57.746625 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.746489    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 19:29:57.746625 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.746515    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 19:29:57.746625 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.746537    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 19:29:57.761784 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.761766    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"]
Sep 11 19:29:57.761927 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.761906    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:57.764552 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.764535    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 19:29:57.764753 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.764739    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 19:29:57.764814 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.764782    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 19:29:57.764860 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.764824    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 19:29:57.780454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.780437    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bf7847486-ccpsb"]
Sep 11 19:29:57.780532 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.780465    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"]
Sep 11 19:29:57.780532 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.780478    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5wg6v"]
Sep 11 19:29:57.780595 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.780549    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:29:57.783113 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.783081    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f4pk6\""
Sep 11 19:29:57.783185 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.783139    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 19:29:57.792637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.792619    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"]
Sep 11 19:29:57.792701 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.792647    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tghl6"]
Sep 11 19:29:57.792740 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.792715    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:57.795873 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.795729    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 19:29:57.796598 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.795978    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 19:29:57.796598 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.796238    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6smxv\""
Sep 11 19:29:57.809203 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.809177    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"]
Sep 11 19:29:57.809203 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.809203    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tghl6"]
Sep 11 19:29:57.809349 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.809215    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5wg6v"]
Sep 11 19:29:57.809349 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.809295    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tghl6"
Sep 11 19:29:57.811805 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.811771    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dplr6\""
Sep 11 19:29:57.811886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.811824    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 19:29:57.811886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.811838    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 19:29:57.811886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.811866    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 19:29:57.823379 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823360    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-certificates\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.823474 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823384    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-bound-sa-token\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.823474 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823413    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-ca-trust-extracted\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.823563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823472    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-image-registry-private-configuration\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.823563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823513    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-trusted-ca\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.823563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v5rn\" (UniqueName: \"kubernetes.io/projected/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-kube-api-access-9v5rn\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.823662 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823573    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-installation-pull-secrets\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.823695 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrj9\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-kube-api-access-vkrj9\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.823727 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823693    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.823768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-tmp\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.823805 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.823770    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.867798 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.867773    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:29:57.870730 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.870707    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7w7kq\""
Sep 11 19:29:57.870730 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.870718    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 19:29:57.924113 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924071    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-ca-trust-extracted\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.924233 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-image-registry-private-configuration\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.924233 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924170    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-trusted-ca\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.924233 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:57.924233 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb80e9a8-e050-4d66-9313-2495201b3238-tmp-dir\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:57.924456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924247    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5006eaf-06a5-4489-a487-826bf6640a1b-cert\") pod \"ingress-canary-tghl6\" (UID: \"b5006eaf-06a5-4489-a487-826bf6640a1b\") " pod="openshift-ingress-canary/ingress-canary-tghl6"
Sep 11 19:29:57.924456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:57.924456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcqn\" (UniqueName: \"kubernetes.io/projected/eb80e9a8-e050-4d66-9313-2495201b3238-kube-api-access-brcqn\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:57.924456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ztm\" (UniqueName: \"kubernetes.io/projected/6ae293b8-32ac-4b05-8dea-fe74ee042b07-kube-api-access-95ztm\") pod \"managed-serviceaccount-addon-agent-7668d66d58-72spw\" (UID: \"6ae293b8-32ac-4b05-8dea-fe74ee042b07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:29:57.924456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v5rn\" (UniqueName: \"kubernetes.io/projected/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-kube-api-access-9v5rn\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.924456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924429    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-installation-pull-secrets\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924500    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrj9\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-kube-api-access-vkrj9\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924524    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-hub\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924526    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-ca-trust-extracted\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924541    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ae293b8-32ac-4b05-8dea-fe74ee042b07-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7668d66d58-72spw\" (UID: \"6ae293b8-32ac-4b05-8dea-fe74ee042b07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924560    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-tmp\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924617    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb80e9a8-e050-4d66-9313-2495201b3238-config-volume\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdcs\" (UniqueName: \"kubernetes.io/projected/b5006eaf-06a5-4489-a487-826bf6640a1b-kube-api-access-9fdcs\") pod \"ingress-canary-tghl6\" (UID: \"b5006eaf-06a5-4489-a487-826bf6640a1b\") " pod="openshift-ingress-canary/ingress-canary-tghl6"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924716    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-ca\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:57.924769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924761    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-certificates\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924845    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-bound-sa-token\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2778046-0e46-405e-a116-bb31fa413068-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924886    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-tmp\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.924938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4t4\" (UniqueName: \"kubernetes.io/projected/f2778046-0e46-405e-a116-bb31fa413068-kube-api-access-nv4t4\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.925127    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.925141    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:57.925236    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:58.425214854 +0000 UTC m=+35.101151743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:29:57.925303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.925296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-trusted-ca\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.925754 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.925416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-certificates\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.929171 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.929075    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-image-registry-private-configuration\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.929285 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.929250    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:57.929600 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.929579    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-installation-pull-secrets\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.936062 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.936015    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-bound-sa-token\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.936240 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.936218    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrj9\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-kube-api-access-vkrj9\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:57.936240 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:57.936233    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v5rn\" (UniqueName: \"kubernetes.io/projected/0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c-kube-api-access-9v5rn\") pod \"klusterlet-addon-workmgr-56c7f9fbff-rtl5t\" (UID: \"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:58.026292 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:58.026292 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026259    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb80e9a8-e050-4d66-9313-2495201b3238-tmp-dir\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:58.026292 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5006eaf-06a5-4489-a487-826bf6640a1b-cert\") pod \"ingress-canary-tghl6\" (UID: \"b5006eaf-06a5-4489-a487-826bf6640a1b\") " pod="openshift-ingress-canary/ingress-canary-tghl6"
Sep 11 19:29:58.026292 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026289    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.026621 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026317    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcqn\" (UniqueName: \"kubernetes.io/projected/eb80e9a8-e050-4d66-9313-2495201b3238-kube-api-access-brcqn\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:58.026621 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:58.026393    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:29:58.026621 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:58.026485    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:58.526463372 +0000 UTC m=+35.202400273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:29:58.026621 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ztm\" (UniqueName: \"kubernetes.io/projected/6ae293b8-32ac-4b05-8dea-fe74ee042b07-kube-api-access-95ztm\") pod \"managed-serviceaccount-addon-agent-7668d66d58-72spw\" (UID: \"6ae293b8-32ac-4b05-8dea-fe74ee042b07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:29:58.026621 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026612    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb80e9a8-e050-4d66-9313-2495201b3238-tmp-dir\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:58.026879 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.026879 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026685    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-hub\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.026879 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ae293b8-32ac-4b05-8dea-fe74ee042b07-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7668d66d58-72spw\" (UID: \"6ae293b8-32ac-4b05-8dea-fe74ee042b07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:29:58.026879 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026773    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb80e9a8-e050-4d66-9313-2495201b3238-config-volume\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:58.027084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdcs\" (UniqueName: \"kubernetes.io/projected/b5006eaf-06a5-4489-a487-826bf6640a1b-kube-api-access-9fdcs\") pod \"ingress-canary-tghl6\" (UID: \"b5006eaf-06a5-4489-a487-826bf6640a1b\") " pod="openshift-ingress-canary/ingress-canary-tghl6"
Sep 11 19:29:58.027084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.026937    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-ca\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.027084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.027025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2778046-0e46-405e-a116-bb31fa413068-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.027084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.027052    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4t4\" (UniqueName: \"kubernetes.io/projected/f2778046-0e46-405e-a116-bb31fa413068-kube-api-access-nv4t4\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.027306 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.027290    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb80e9a8-e050-4d66-9313-2495201b3238-config-volume\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:58.027850 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.027822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2778046-0e46-405e-a116-bb31fa413068-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.029701 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.029673    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.030168 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.030148    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5006eaf-06a5-4489-a487-826bf6640a1b-cert\") pod \"ingress-canary-tghl6\" (UID: \"b5006eaf-06a5-4489-a487-826bf6640a1b\") " pod="openshift-ingress-canary/ingress-canary-tghl6"
Sep 11 19:29:58.030257 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.030243    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-ca\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.030324 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.030294    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-hub\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.030324 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.030317    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ae293b8-32ac-4b05-8dea-fe74ee042b07-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7668d66d58-72spw\" (UID: \"6ae293b8-32ac-4b05-8dea-fe74ee042b07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:29:58.030492 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.030333    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2778046-0e46-405e-a116-bb31fa413068-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.035116 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.035074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcqn\" (UniqueName: \"kubernetes.io/projected/eb80e9a8-e050-4d66-9313-2495201b3238-kube-api-access-brcqn\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:58.035433 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.035412    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ztm\" (UniqueName: \"kubernetes.io/projected/6ae293b8-32ac-4b05-8dea-fe74ee042b07-kube-api-access-95ztm\") pod \"managed-serviceaccount-addon-agent-7668d66d58-72spw\" (UID: \"6ae293b8-32ac-4b05-8dea-fe74ee042b07\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:29:58.035880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.035858    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4t4\" (UniqueName: \"kubernetes.io/projected/f2778046-0e46-405e-a116-bb31fa413068-kube-api-access-nv4t4\") pod \"cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6\" (UID: \"f2778046-0e46-405e-a116-bb31fa413068\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.036182 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.036092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdcs\" (UniqueName: \"kubernetes.io/projected/b5006eaf-06a5-4489-a487-826bf6640a1b-kube-api-access-9fdcs\") pod \"ingress-canary-tghl6\" (UID: \"b5006eaf-06a5-4489-a487-826bf6640a1b\") " pod="openshift-ingress-canary/ingress-canary-tghl6"
Sep 11 19:29:58.052016 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.051997    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:29:58.084650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.084619    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:29:58.090763 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.090739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:29:58.117970 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.117943    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tghl6"
Sep 11 19:29:58.431609 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.431569    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:58.431878 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:58.431744    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:29:58.431878 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:58.431767    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:29:58.431878 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:58.431839    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:59.431816675 +0000 UTC m=+36.107753574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:29:58.533063 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.533027    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:58.533267 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:58.533217    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:29:58.533321 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:58.533288    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:59.533270962 +0000 UTC m=+36.209207848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:29:58.867748 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.867671    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:29:58.868004 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.867857    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:29:58.870510 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.870484    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 19:29:58.870727 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.870706    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 19:29:58.870798 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.870743    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 19:29:58.870798 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:58.870772    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2xtcq\""
Sep 11 19:29:59.439547 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:59.439512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:29:59.439706 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:59.439671    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:29:59.439706 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:59.439691    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:29:59.439815 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:59.439743    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:01.43972497 +0000 UTC m=+38.115661855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:29:59.541028 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:59.540847    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:29:59.541201 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:59.540967    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:29:59.541201 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:29:59.541088    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:01.541073143 +0000 UTC m=+38.217010029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:29:59.681545 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:59.681521    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"]
Sep 11 19:29:59.685408 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:59.685236    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ae293b8_32ac_4b05_8dea_fe74ee042b07.slice/crio-7abb11f2c61d6a30d6a8b4536b958c0b1c2abaf5a2c0f3fbb7ef9094f8be3724 WatchSource:0}: Error finding container 7abb11f2c61d6a30d6a8b4536b958c0b1c2abaf5a2c0f3fbb7ef9094f8be3724: Status 404 returned error can't find the container with id 7abb11f2c61d6a30d6a8b4536b958c0b1c2abaf5a2c0f3fbb7ef9094f8be3724
Sep 11 19:29:59.686538 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:59.686500    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"]
Sep 11 19:29:59.687714 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:59.687693    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"]
Sep 11 19:29:59.688128 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:59.688075    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5006eaf_06a5_4489_a487_826bf6640a1b.slice/crio-a53ad5841063300ba36970799eda3c6b6dc42aba854e364c99204f32a3aa7133 WatchSource:0}: Error finding container a53ad5841063300ba36970799eda3c6b6dc42aba854e364c99204f32a3aa7133: Status 404 returned error can't find the container with id a53ad5841063300ba36970799eda3c6b6dc42aba854e364c99204f32a3aa7133
Sep 11 19:29:59.689060 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:29:59.688868    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tghl6"]
Sep 11 19:29:59.689242 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:59.689097    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7b79ab_d2aa_4fe5_887f_4f7968b27d4c.slice/crio-a14f5b9defdd428f08a04293a21a674d8f4d2eb587e9663073e637e3b9c47d84 WatchSource:0}: Error finding container a14f5b9defdd428f08a04293a21a674d8f4d2eb587e9663073e637e3b9c47d84: Status 404 returned error can't find the container with id a14f5b9defdd428f08a04293a21a674d8f4d2eb587e9663073e637e3b9c47d84
Sep 11 19:29:59.689808 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:29:59.689741    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2778046_0e46_405e_a116_bb31fa413068.slice/crio-ef23c03e4d9655a099f6d908521ca24c47d0ceafd0641c847f36a6bbcf2cc4ba WatchSource:0}: Error finding container ef23c03e4d9655a099f6d908521ca24c47d0ceafd0641c847f36a6bbcf2cc4ba: Status 404 returned error can't find the container with id ef23c03e4d9655a099f6d908521ca24c47d0ceafd0641c847f36a6bbcf2cc4ba
Sep 11 19:30:00.044141 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:00.044053    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" event={"ID":"6ae293b8-32ac-4b05-8dea-fe74ee042b07","Type":"ContainerStarted","Data":"7abb11f2c61d6a30d6a8b4536b958c0b1c2abaf5a2c0f3fbb7ef9094f8be3724"}
Sep 11 19:30:00.045173 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:00.045154    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" event={"ID":"f2778046-0e46-405e-a116-bb31fa413068","Type":"ContainerStarted","Data":"ef23c03e4d9655a099f6d908521ca24c47d0ceafd0641c847f36a6bbcf2cc4ba"}
Sep 11 19:30:00.046146 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:00.046123    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tghl6" event={"ID":"b5006eaf-06a5-4489-a487-826bf6640a1b","Type":"ContainerStarted","Data":"a53ad5841063300ba36970799eda3c6b6dc42aba854e364c99204f32a3aa7133"}
Sep 11 19:30:00.047144 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:00.047123    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" event={"ID":"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c","Type":"ContainerStarted","Data":"a14f5b9defdd428f08a04293a21a674d8f4d2eb587e9663073e637e3b9c47d84"}
Sep 11 19:30:00.049480 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:00.049459    2571 generic.go:356] "Generic (PLEG): container finished" podID="fb06fd55-d2a4-46c2-89e8-9247da15014f" containerID="88427ce738f85562d9befb42a47e3cea456ed99a393a3f6c218a5488e1d5e110" exitCode=0
Sep 11 19:30:00.049559 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:00.049485    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" event={"ID":"fb06fd55-d2a4-46c2-89e8-9247da15014f","Type":"ContainerDied","Data":"88427ce738f85562d9befb42a47e3cea456ed99a393a3f6c218a5488e1d5e110"}
Sep 11 19:30:01.060843 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:01.060681    2571 generic.go:356] "Generic (PLEG): container finished" podID="fb06fd55-d2a4-46c2-89e8-9247da15014f" containerID="3a02e81f08d979333c9a9a332b252003c2b83c3dfc5368a922ebc0eeba6feb21" exitCode=0
Sep 11 19:30:01.060843 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:01.060764    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" event={"ID":"fb06fd55-d2a4-46c2-89e8-9247da15014f","Type":"ContainerDied","Data":"3a02e81f08d979333c9a9a332b252003c2b83c3dfc5368a922ebc0eeba6feb21"}
Sep 11 19:30:01.456585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:01.456537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:30:01.456763 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:01.456719    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:30:01.456763 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:01.456736    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:30:01.456852 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:01.456795    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:05.456776466 +0000 UTC m=+42.132713363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:30:01.557247 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:01.557212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:30:01.557391 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:01.557364    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:30:01.557458 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:01.557425    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:05.557405889 +0000 UTC m=+42.233342798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:30:02.069519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:02.069480    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" event={"ID":"fb06fd55-d2a4-46c2-89e8-9247da15014f","Type":"ContainerStarted","Data":"11da1554e7c0377639f430a607b52e5acdb6063c3642e66d25edc797bf2c8c1d"}
Sep 11 19:30:02.097061 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:02.097006    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mk9z2" podStartSLOduration=7.150654657 podStartE2EDuration="38.09698511s" totalImagesPullingTime="30.946330453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:02.095698217 +0000 UTC m=+38.771635151" watchObservedRunningTime="2026-09-11 19:30:02.09698511 +0000 UTC m=+38.772922019"
Sep 11 19:30:05.491279 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:05.491240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:30:05.491641 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:05.491401    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:30:05.491641 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:05.491423    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:30:05.491641 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:05.491477    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:13.491462986 +0000 UTC m=+50.167399890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:30:05.592797 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:05.592755    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:30:05.592958 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:05.592910    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:30:05.593004 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:05.592982    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:13.592963393 +0000 UTC m=+50.268900292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:30:07.082793 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:07.082757    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" event={"ID":"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c","Type":"ContainerStarted","Data":"662fac8204fb42d16caab67bc60b6c9e4dc2a08762c31f8ad192c76c5a223598"}
Sep 11 19:30:07.083236 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:07.082926    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:30:07.084437 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:07.084411    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" event={"ID":"6ae293b8-32ac-4b05-8dea-fe74ee042b07","Type":"ContainerStarted","Data":"ac8857b9fa9987636f54326b7ccbe1e59d3cc679302b39565400729a2778bf13"}
Sep 11 19:30:07.084804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:07.084784    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:30:07.085987 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:07.085965    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tghl6" event={"ID":"b5006eaf-06a5-4489-a487-826bf6640a1b","Type":"ContainerStarted","Data":"071b62b91bfba7a3a4833d4c0eb2e9dd3fe725cb7ac482edf94a7442713e32dd"}
Sep 11 19:30:07.098506 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:07.098440    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" podStartSLOduration=4.291695065 podStartE2EDuration="11.098425033s" totalImagesPullingTime="6.806729968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:07.098167133 +0000 UTC m=+43.774104041" watchObservedRunningTime="2026-09-11 19:30:07.098425033 +0000 UTC m=+43.774361942"
Sep 11 19:30:07.113805 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:07.113764    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" podStartSLOduration=4.296946846 podStartE2EDuration="11.113751509s" totalImagesPullingTime="6.816804663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:07.112833862 +0000 UTC m=+43.788770769" watchObservedRunningTime="2026-09-11 19:30:07.113751509 +0000 UTC m=+43.789688417"
Sep 11 19:30:07.131848 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:07.129938    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tghl6" podStartSLOduration=3.322336044 podStartE2EDuration="10.129919895s" totalImagesPullingTime="6.807583851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:07.127985068 +0000 UTC m=+43.803921976" watchObservedRunningTime="2026-09-11 19:30:07.129919895 +0000 UTC m=+43.805856806"
Sep 11 19:30:11.097933 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:11.097904    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" event={"ID":"f2778046-0e46-405e-a116-bb31fa413068","Type":"ContainerStarted","Data":"a72e8994bf5c4788fa9905018d9d5d7e91c32531f1bfa0ef2da56a9452b1d200"}
Sep 11 19:30:12.102547 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:12.102511    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" event={"ID":"f2778046-0e46-405e-a116-bb31fa413068","Type":"ContainerStarted","Data":"26407a5286b98ab20564cf125903d4803b94e466ac2881086f7cc16ce86f5f45"}
Sep 11 19:30:12.102928 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:12.102560    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" event={"ID":"f2778046-0e46-405e-a116-bb31fa413068","Type":"ContainerStarted","Data":"58e58ed4141baece752470f9dc08a5665deb7b1323af0d8b310dcddd4ca0cddf"}
Sep 11 19:30:12.121950 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:12.121887    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" podStartSLOduration=4.909804999 podStartE2EDuration="16.121868227s" totalImagesPullingTime="11.212063228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:12.121802849 +0000 UTC m=+48.797739757" watchObservedRunningTime="2026-09-11 19:30:12.121868227 +0000 UTC m=+48.797805132"
Sep 11 19:30:13.555178 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:13.555134    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:30:13.555589 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:13.555283    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:30:13.555589 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:13.555302    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:30:13.555589 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:13.555360    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:29.555343996 +0000 UTC m=+66.231280901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:30:13.656040 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:13.656002    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:30:13.656192 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:13.656166    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:30:13.656250 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:13.656240    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:29.656223737 +0000 UTC m=+66.332160628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:30:23.030150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:23.030121    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pj8nm"
Sep 11 19:30:29.467375 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.467328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:30:29.467375 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.467373    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:30:29.470715 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.470689    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 19:30:29.470778 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.470713    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 19:30:29.478029 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:29.478003    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 19:30:29.478138 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:29.478095    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:31:33.478077391 +0000 UTC m=+130.154014281 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : secret "metrics-daemon-secret" not found
Sep 11 19:30:29.480324 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.480305    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 19:30:29.491254 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.491233    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6gdc\" (UniqueName: \"kubernetes.io/projected/a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d-kube-api-access-k6gdc\") pod \"network-check-target-gs762\" (UID: \"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d\") " pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:30:29.568542 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.568505    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:30:29.568542 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.568543    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:30:29.568784 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:29.568710    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:30:29.568784 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:29.568734    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:30:29.568888 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:29.568815    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:31:01.56879317 +0000 UTC m=+98.244730056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:30:29.571690 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.571669    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 19:30:29.581902 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.581868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/412e8257-acb6-41f7-80de-b26261cb8fcb-original-pull-secret\") pod \"global-pull-secret-syncer-z292n\" (UID: \"412e8257-acb6-41f7-80de-b26261cb8fcb\") " pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:30:29.669290 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.669249    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:30:29.669480 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:29.669402    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:30:29.669480 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:30:29.669467    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:31:01.669452293 +0000 UTC m=+98.345389179 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:30:29.781601 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.781514    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2xtcq\""
Sep 11 19:30:29.784910 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.784891    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z292n"
Sep 11 19:30:29.789031 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:29.789013    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:30:30.060199 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:30.060089    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z292n"]
Sep 11 19:30:30.061027 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:30.060965    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gs762"]
Sep 11 19:30:30.063001 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:30:30.062972    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412e8257_acb6_41f7_80de_b26261cb8fcb.slice/crio-95abd9584d667f3cdacfd0b601284e8cd9e5d6f9d3014efb4b8dcf68301c9099 WatchSource:0}: Error finding container 95abd9584d667f3cdacfd0b601284e8cd9e5d6f9d3014efb4b8dcf68301c9099: Status 404 returned error can't find the container with id 95abd9584d667f3cdacfd0b601284e8cd9e5d6f9d3014efb4b8dcf68301c9099
Sep 11 19:30:30.063587 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:30:30.063568    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ee76b5_026d_45bd_9cbf_0d9e4c144e0d.slice/crio-0347c1b3dee69072d500c6fe75a7956d62cf25bb5edd01d2e67d983fdae2fac4 WatchSource:0}: Error finding container 0347c1b3dee69072d500c6fe75a7956d62cf25bb5edd01d2e67d983fdae2fac4: Status 404 returned error can't find the container with id 0347c1b3dee69072d500c6fe75a7956d62cf25bb5edd01d2e67d983fdae2fac4
Sep 11 19:30:30.157215 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:30.157178    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z292n" event={"ID":"412e8257-acb6-41f7-80de-b26261cb8fcb","Type":"ContainerStarted","Data":"95abd9584d667f3cdacfd0b601284e8cd9e5d6f9d3014efb4b8dcf68301c9099"}
Sep 11 19:30:30.160078 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:30.160052    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gs762" event={"ID":"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d","Type":"ContainerStarted","Data":"0347c1b3dee69072d500c6fe75a7956d62cf25bb5edd01d2e67d983fdae2fac4"}
Sep 11 19:30:36.182444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:36.182403    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z292n" event={"ID":"412e8257-acb6-41f7-80de-b26261cb8fcb","Type":"ContainerStarted","Data":"762140e1066c1d83b7504c54b0e01bdf720ce1bbaa1c34e3859c61c0cfade758"}
Sep 11 19:30:36.183951 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:36.183920    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gs762" event={"ID":"a7ee76b5-026d-45bd-9cbf-0d9e4c144e0d","Type":"ContainerStarted","Data":"f406801c03d6e3e5afeb4bbf0e0a4986e9ac7c2855841a781c083e7b6a92ade6"}
Sep 11 19:30:36.184062 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:36.183981    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:30:36.200032 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:36.199983    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z292n" podStartSLOduration=66.729034058 podStartE2EDuration="1m12.19996486s" totalImagesPullingTime="5.470930802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:36.199316434 +0000 UTC m=+72.875253343" watchObservedRunningTime="2026-09-11 19:30:36.19996486 +0000 UTC m=+72.875901765"
Sep 11 19:30:36.214969 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:30:36.214926    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gs762" podStartSLOduration=66.755576314 podStartE2EDuration="1m12.214912031s" totalImagesPullingTime="5.459335717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:36.21466488 +0000 UTC m=+72.890601789" watchObservedRunningTime="2026-09-11 19:30:36.214912031 +0000 UTC m=+72.890848939"
Sep 11 19:31:01.602945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:31:01.602852    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:31:01.603328 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:31:01.603004    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:31:01.603328 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:31:01.603019    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:31:01.603328 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:31:01.603079    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:32:05.603062542 +0000 UTC m=+162.278999440 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:31:01.703634 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:31:01.703587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:31:01.703826 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:31:01.703737    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:31:01.703826 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:31:01.703807    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:32:05.703789141 +0000 UTC m=+162.379726033 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:31:07.191405 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:31:07.191374    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gs762"
Sep 11 19:31:33.541607 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:31:33.541556    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:31:33.542040 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:31:33.541694    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 19:31:33.542040 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:31:33.541754    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:35.541737996 +0000 UTC m=+252.217674887 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : secret "metrics-daemon-secret" not found
Sep 11 19:32:00.741062 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:32:00.741013    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" podUID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"
Sep 11 19:32:00.811960 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:32:00.811906    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5wg6v" podUID="eb80e9a8-e050-4d66-9313-2495201b3238"
Sep 11 19:32:00.878057 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:32:00.878018    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:32:01.425427 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:01.425396    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:32:01.425427 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:01.425427    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:32:05.670657 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:05.670618    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:32:05.671066 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:32:05.670769    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:32:05.671066 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:32:05.670788    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7847486-ccpsb: secret "image-registry-tls" not found
Sep 11 19:32:05.671066 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:32:05.670869    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls podName:a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50 nodeName:}" failed. No retries permitted until 2026-09-11 19:34:07.670847957 +0000 UTC m=+284.346784847 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls") pod "image-registry-5bf7847486-ccpsb" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50") : secret "image-registry-tls" not found
Sep 11 19:32:05.771627 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:05.771588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:32:05.771790 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:32:05.771714    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:32:05.771790 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:32:05.771784    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls podName:eb80e9a8-e050-4d66-9313-2495201b3238 nodeName:}" failed. No retries permitted until 2026-09-11 19:34:07.771766702 +0000 UTC m=+284.447703594 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls") pod "dns-default-5wg6v" (UID: "eb80e9a8-e050-4d66-9313-2495201b3238") : secret "dns-default-metrics-tls" not found
Sep 11 19:32:07.083629 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:07.083529    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" podUID="0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
Sep 11 19:32:07.443876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:07.443844    2571 generic.go:356] "Generic (PLEG): container finished" podID="6ae293b8-32ac-4b05-8dea-fe74ee042b07" containerID="ac8857b9fa9987636f54326b7ccbe1e59d3cc679302b39565400729a2778bf13" exitCode=255
Sep 11 19:32:07.444051 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:07.443918    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" event={"ID":"6ae293b8-32ac-4b05-8dea-fe74ee042b07","Type":"ContainerDied","Data":"ac8857b9fa9987636f54326b7ccbe1e59d3cc679302b39565400729a2778bf13"}
Sep 11 19:32:07.444284 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:07.444270    2571 scope.go:117] "RemoveContainer" containerID="ac8857b9fa9987636f54326b7ccbe1e59d3cc679302b39565400729a2778bf13"
Sep 11 19:32:07.445556 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:07.445521    2571 generic.go:356] "Generic (PLEG): container finished" podID="0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c" containerID="662fac8204fb42d16caab67bc60b6c9e4dc2a08762c31f8ad192c76c5a223598" exitCode=1
Sep 11 19:32:07.445615 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:07.445560    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" event={"ID":"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c","Type":"ContainerDied","Data":"662fac8204fb42d16caab67bc60b6c9e4dc2a08762c31f8ad192c76c5a223598"}
Sep 11 19:32:07.445860 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:07.445845    2571 scope.go:117] "RemoveContainer" containerID="662fac8204fb42d16caab67bc60b6c9e4dc2a08762c31f8ad192c76c5a223598"
Sep 11 19:32:08.053211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:08.053176    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:32:08.091119 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:08.091076    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:32:08.449923 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:08.449884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" event={"ID":"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c","Type":"ContainerStarted","Data":"5b3911dbc180448bd012be8a30028e41611e18668dc25de818216c91dd04ece1"}
Sep 11 19:32:08.450129 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:08.450080    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:32:08.451066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:08.451045    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:32:08.451779 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:08.451760    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" event={"ID":"6ae293b8-32ac-4b05-8dea-fe74ee042b07","Type":"ContainerStarted","Data":"c42fff6bf31fcc41897dba9187573e967677539222fca9fe97c57db9061595a3"}
Sep 11 19:32:12.867821 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:32:12.867784    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:33:35.615049 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:35.615010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:33:35.615558 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:35.615155    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 19:33:35.615558 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:35.615209    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs podName:d81d2d85-34c6-44f9-a0e4-17ae596fea59 nodeName:}" failed. No retries permitted until 2026-09-11 19:35:37.615195275 +0000 UTC m=+374.291132160 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs") pod "network-metrics-daemon-mtpxz" (UID: "d81d2d85-34c6-44f9-a0e4-17ae596fea59") : secret "metrics-daemon-secret" not found
Sep 11 19:33:40.490607 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.490569    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l8fhz"]
Sep 11 19:33:40.493716 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.493692    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.496434 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.496414    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 19:33:40.496531 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.496414    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 19:33:40.496608 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.496591    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vrbnh\""
Sep 11 19:33:40.496680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.496663    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 19:33:40.497620 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.497602    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 19:33:40.503845 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.503827    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8fhz"]
Sep 11 19:33:40.550676 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.550640    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.550851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.550706    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98ef2ecc-f11a-40e7-9982-cb00464b38d4-data-volume\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.550851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.550744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98ef2ecc-f11a-40e7-9982-cb00464b38d4-crio-socket\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.550851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.550815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl85q\" (UniqueName: \"kubernetes.io/projected/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-api-access-bl85q\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.550978 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.550874    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.651298 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.651267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98ef2ecc-f11a-40e7-9982-cb00464b38d4-crio-socket\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.651469 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.651317    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl85q\" (UniqueName: \"kubernetes.io/projected/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-api-access-bl85q\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.651469 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.651352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.651469 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.651369    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.651469 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.651399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98ef2ecc-f11a-40e7-9982-cb00464b38d4-data-volume\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.651469 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.651400    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98ef2ecc-f11a-40e7-9982-cb00464b38d4-crio-socket\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.651469 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:40.651462    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:33:40.651741 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:40.651508    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:40.651741 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:40.651517    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:41.151502393 +0000 UTC m=+257.827439280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : configmap "kube-rbac-proxy" not found
Sep 11 19:33:40.651741 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:40.651577    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:41.151557166 +0000 UTC m=+257.827494053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:40.651741 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.651669    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98ef2ecc-f11a-40e7-9982-cb00464b38d4-data-volume\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:40.659192 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:40.659173    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl85q\" (UniqueName: \"kubernetes.io/projected/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-api-access-bl85q\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:41.155248 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:41.155204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:41.155248 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:41.155250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:41.155586 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:41.155337    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:33:41.155586 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:41.155353    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:41.155586 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:41.155388    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:42.15537555 +0000 UTC m=+258.831312441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : configmap "kube-rbac-proxy" not found
Sep 11 19:33:41.155586 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:41.155415    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:42.155401002 +0000 UTC m=+258.831337888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:42.164469 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:42.164431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:42.164469 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:42.164469    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:42.164910 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:42.164576    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:33:42.164910 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:42.164628    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:44.164616003 +0000 UTC m=+260.840552888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : configmap "kube-rbac-proxy" not found
Sep 11 19:33:42.164910 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:42.164575    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:42.164910 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:42.164703    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:44.164691006 +0000 UTC m=+260.840627905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:43.463045 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:43.463016    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wmckq_25cf9814-dfe6-41da-b125-56e077f8faf8/dns-node-resolver/0.log"
Sep 11 19:33:44.180218 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:44.180159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:44.180218 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:44.180212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:44.180455 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:44.180331    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:33:44.180455 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:44.180349    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:44.180455 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:44.180401    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:48.180378026 +0000 UTC m=+264.856314911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : configmap "kube-rbac-proxy" not found
Sep 11 19:33:44.180455 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:44.180424    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:48.180409486 +0000 UTC m=+264.856346372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:44.463910 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:44.463840    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xg2gb_c6b1a476-e4f1-4aa3-9252-3cffe7e7e903/node-ca/0.log"
Sep 11 19:33:45.262653 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:45.262623    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tghl6_b5006eaf-06a5-4489-a487-826bf6640a1b/serve-healthcheck-canary/0.log"
Sep 11 19:33:48.211461 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:48.211376    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:48.211461 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:48.211414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:48.211937 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:48.211528    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:48.211937 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:48.211565    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:33:48.211937 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:48.211582    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:56.211569645 +0000 UTC m=+272.887506536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:33:48.211937 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:33:48.211622    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm podName:98ef2ecc-f11a-40e7-9982-cb00464b38d4 nodeName:}" failed. No retries permitted until 2026-09-11 19:33:56.211609242 +0000 UTC m=+272.887546132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l8fhz" (UID: "98ef2ecc-f11a-40e7-9982-cb00464b38d4") : configmap "kube-rbac-proxy" not found
Sep 11 19:33:56.271176 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:56.271136    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:56.271176 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:56.271179    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:56.271842 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:56.271818    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98ef2ecc-f11a-40e7-9982-cb00464b38d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:56.273374 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:56.273353    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98ef2ecc-f11a-40e7-9982-cb00464b38d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8fhz\" (UID: \"98ef2ecc-f11a-40e7-9982-cb00464b38d4\") " pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:56.402194 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:56.402150    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8fhz"
Sep 11 19:33:56.668356 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:56.668318    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8fhz"]
Sep 11 19:33:56.671886 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:33:56.671855    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ef2ecc_f11a_40e7_9982_cb00464b38d4.slice/crio-027299fc0b02f321a80f43ca1348bcb64d32a8c512c3d24c964b335074927c76 WatchSource:0}: Error finding container 027299fc0b02f321a80f43ca1348bcb64d32a8c512c3d24c964b335074927c76: Status 404 returned error can't find the container with id 027299fc0b02f321a80f43ca1348bcb64d32a8c512c3d24c964b335074927c76
Sep 11 19:33:56.769236 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:56.769199    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8fhz" event={"ID":"98ef2ecc-f11a-40e7-9982-cb00464b38d4","Type":"ContainerStarted","Data":"24ac7c9ba1388f726858f8d2ed857f70cdcea110af19c432a2c0ca392a3a5ef3"}
Sep 11 19:33:56.769383 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:56.769270    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8fhz" event={"ID":"98ef2ecc-f11a-40e7-9982-cb00464b38d4","Type":"ContainerStarted","Data":"027299fc0b02f321a80f43ca1348bcb64d32a8c512c3d24c964b335074927c76"}
Sep 11 19:33:57.776207 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:57.776125    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8fhz" event={"ID":"98ef2ecc-f11a-40e7-9982-cb00464b38d4","Type":"ContainerStarted","Data":"ef42abaafd48eae153557a72b203199f8aee5833fe00129ffa5c17370049bfdd"}
Sep 11 19:33:59.784853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:59.784819    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8fhz" event={"ID":"98ef2ecc-f11a-40e7-9982-cb00464b38d4","Type":"ContainerStarted","Data":"fe4be268b5be676197392033e6bfe9790fecdd28fe21abf62125558ddd0e4be9"}
Sep 11 19:33:59.806688 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:33:59.806635    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l8fhz" podStartSLOduration=17.74655197 podStartE2EDuration="19.806620584s" totalImagesPullingTime="2.060068614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:33:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:33:59.804781958 +0000 UTC m=+276.480718866" watchObservedRunningTime="2026-09-11 19:33:59.806620584 +0000 UTC m=+276.482557492"
Sep 11 19:34:04.426353 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:04.426290    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" podUID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"
Sep 11 19:34:04.426353 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:04.426290    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5wg6v" podUID="eb80e9a8-e050-4d66-9313-2495201b3238"
Sep 11 19:34:04.800537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:04.800456    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:34:04.800674 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:04.800456    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:34:07.758236 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.758211    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:34:07.760290 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.760272    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"image-registry-5bf7847486-ccpsb\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") " pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:34:07.804450 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.804401    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5wflx\""
Sep 11 19:34:07.811405 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.811390    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:34:07.811702 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.811681    2571 generic.go:356] "Generic (PLEG): container finished" podID="6ae293b8-32ac-4b05-8dea-fe74ee042b07" containerID="c42fff6bf31fcc41897dba9187573e967677539222fca9fe97c57db9061595a3" exitCode=255
Sep 11 19:34:07.811776 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.811757    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" event={"ID":"6ae293b8-32ac-4b05-8dea-fe74ee042b07","Type":"ContainerDied","Data":"c42fff6bf31fcc41897dba9187573e967677539222fca9fe97c57db9061595a3"}
Sep 11 19:34:07.811820 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.811806    2571 scope.go:117] "RemoveContainer" containerID="ac8857b9fa9987636f54326b7ccbe1e59d3cc679302b39565400729a2778bf13"
Sep 11 19:34:07.812081 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.812065    2571 scope.go:117] "RemoveContainer" containerID="c42fff6bf31fcc41897dba9187573e967677539222fca9fe97c57db9061595a3"
Sep 11 19:34:07.812277 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:07.812249    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7668d66d58-72spw_open-cluster-management-agent-addon(6ae293b8-32ac-4b05-8dea-fe74ee042b07)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" podUID="6ae293b8-32ac-4b05-8dea-fe74ee042b07"
Sep 11 19:34:07.813627 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.813606    2571 generic.go:356] "Generic (PLEG): container finished" podID="0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c" containerID="5b3911dbc180448bd012be8a30028e41611e18668dc25de818216c91dd04ece1" exitCode=1
Sep 11 19:34:07.813709 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.813636    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" event={"ID":"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c","Type":"ContainerDied","Data":"5b3911dbc180448bd012be8a30028e41611e18668dc25de818216c91dd04ece1"}
Sep 11 19:34:07.813939 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.813917    2571 scope.go:117] "RemoveContainer" containerID="5b3911dbc180448bd012be8a30028e41611e18668dc25de818216c91dd04ece1"
Sep 11 19:34:07.814052 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:07.814038    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-56c7f9fbff-rtl5t_open-cluster-management-agent-addon(0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" podUID="0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c"
Sep 11 19:34:07.829754 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.829736    2571 scope.go:117] "RemoveContainer" containerID="662fac8204fb42d16caab67bc60b6c9e4dc2a08762c31f8ad192c76c5a223598"
Sep 11 19:34:07.859127 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.859087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:34:07.862056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:07.862002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb80e9a8-e050-4d66-9313-2495201b3238-metrics-tls\") pod \"dns-default-5wg6v\" (UID: \"eb80e9a8-e050-4d66-9313-2495201b3238\") " pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:34:08.052499 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.052472    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:34:08.080558 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.080505    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bf7847486-ccpsb"]
Sep 11 19:34:08.083608 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:34:08.083584    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10d1ce3_6a6a_44d3_bdd3_676adfa4ac50.slice/crio-169a288fd6b17def3025d66845d5fb708f62b774c890fac2aa4d82003cb19f9a WatchSource:0}: Error finding container 169a288fd6b17def3025d66845d5fb708f62b774c890fac2aa4d82003cb19f9a: Status 404 returned error can't find the container with id 169a288fd6b17def3025d66845d5fb708f62b774c890fac2aa4d82003cb19f9a
Sep 11 19:34:08.091574 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.091556    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw"
Sep 11 19:34:08.103880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.103862    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6smxv\""
Sep 11 19:34:08.111770 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.111747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:34:08.377772 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.377746    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5wg6v"]
Sep 11 19:34:08.380522 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:34:08.380496    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb80e9a8_e050_4d66_9313_2495201b3238.slice/crio-11c106f8e5bd920c5d5f477b8e7edf583c1b848e84f035078a43a808a14c027e WatchSource:0}: Error finding container 11c106f8e5bd920c5d5f477b8e7edf583c1b848e84f035078a43a808a14c027e: Status 404 returned error can't find the container with id 11c106f8e5bd920c5d5f477b8e7edf583c1b848e84f035078a43a808a14c027e
Sep 11 19:34:08.450724 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.450683    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:34:08.818451 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.818353    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" event={"ID":"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50","Type":"ContainerStarted","Data":"e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345"}
Sep 11 19:34:08.818451 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.818413    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" event={"ID":"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50","Type":"ContainerStarted","Data":"169a288fd6b17def3025d66845d5fb708f62b774c890fac2aa4d82003cb19f9a"}
Sep 11 19:34:08.818884 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.818475    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:34:08.820062 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.820042    2571 scope.go:117] "RemoveContainer" containerID="5b3911dbc180448bd012be8a30028e41611e18668dc25de818216c91dd04ece1"
Sep 11 19:34:08.820239 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:08.820211    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-56c7f9fbff-rtl5t_open-cluster-management-agent-addon(0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" podUID="0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c"
Sep 11 19:34:08.821346 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.821329    2571 scope.go:117] "RemoveContainer" containerID="c42fff6bf31fcc41897dba9187573e967677539222fca9fe97c57db9061595a3"
Sep 11 19:34:08.821473 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:08.821455    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7668d66d58-72spw_open-cluster-management-agent-addon(6ae293b8-32ac-4b05-8dea-fe74ee042b07)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" podUID="6ae293b8-32ac-4b05-8dea-fe74ee042b07"
Sep 11 19:34:08.822189 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.822170    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wg6v" event={"ID":"eb80e9a8-e050-4d66-9313-2495201b3238","Type":"ContainerStarted","Data":"11c106f8e5bd920c5d5f477b8e7edf583c1b848e84f035078a43a808a14c027e"}
Sep 11 19:34:08.839305 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:08.839256    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" podStartSLOduration=284.839238476 podStartE2EDuration="4m44.839238476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:08.837535545 +0000 UTC m=+285.513472453" watchObservedRunningTime="2026-09-11 19:34:08.839238476 +0000 UTC m=+285.515175387"
Sep 11 19:34:09.830179 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:09.830149    2571 scope.go:117] "RemoveContainer" containerID="5b3911dbc180448bd012be8a30028e41611e18668dc25de818216c91dd04ece1"
Sep 11 19:34:09.830636 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:09.830339    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-56c7f9fbff-rtl5t_open-cluster-management-agent-addon(0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" podUID="0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c"
Sep 11 19:34:10.835628 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:10.835592    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wg6v" event={"ID":"eb80e9a8-e050-4d66-9313-2495201b3238","Type":"ContainerStarted","Data":"394a316017e591bc5ca321125c2450aa7e44ee60b782ba0c0cdc9a4a58b53a69"}
Sep 11 19:34:10.836008 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:10.835647    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wg6v" event={"ID":"eb80e9a8-e050-4d66-9313-2495201b3238","Type":"ContainerStarted","Data":"8132a01cc9f85e313e3fff49de3a067a13ffc01b0fe4c9b83ff2a7355f0dc2bf"}
Sep 11 19:34:10.836008 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:10.835727    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:34:10.855285 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:10.855240    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5wg6v" podStartSLOduration=252.239840221 podStartE2EDuration="4m13.855227296s" totalImagesPullingTime="1.615387075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:10.854500858 +0000 UTC m=+287.530437766" watchObservedRunningTime="2026-09-11 19:34:10.855227296 +0000 UTC m=+287.531164203"
Sep 11 19:34:15.868467 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:15.868424    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mtpxz" podUID="d81d2d85-34c6-44f9-a0e4-17ae596fea59"
Sep 11 19:34:19.012455 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.012424    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mhbbb"]
Sep 11 19:34:19.015341 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.015324    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.017808 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.017784    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 19:34:19.018164 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.018144    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 19:34:19.019632 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.019601    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 19:34:19.019632 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.019625    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7lgv6\""
Sep 11 19:34:19.019784 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.019636    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 19:34:19.019784 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.019650    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 19:34:19.019784 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.019667    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 19:34:19.135419 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135371    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.135597 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-root\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.135597 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135458    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-wtmp\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.135597 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-metrics-client-ca\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.135597 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135497    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-sys\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.135597 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-textfile\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.135597 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgr77\" (UniqueName: \"kubernetes.io/projected/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-kube-api-access-pgr77\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.135809 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.135809 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.135645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-tls\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236570 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236533    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-sys\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236570 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236576    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-textfile\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236772 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236633    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-sys\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236772 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236696    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgr77\" (UniqueName: \"kubernetes.io/projected/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-kube-api-access-pgr77\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236772 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236772 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236761    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-tls\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236791    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-root\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236883    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-textfile\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.236920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236895    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-wtmp\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.237093 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236936    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-metrics-client-ca\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.237093 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.236999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-root\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.237093 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.237017    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-wtmp\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.237394 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.237359    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.237495 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.237468    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-metrics-client-ca\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.239046 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.239021    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.239206 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.239190    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-node-exporter-tls\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.245287 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.245266    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgr77\" (UniqueName: \"kubernetes.io/projected/5392b55e-78e6-4494-8fbe-f5387f6c4cb3-kube-api-access-pgr77\") pod \"node-exporter-mhbbb\" (UID: \"5392b55e-78e6-4494-8fbe-f5387f6c4cb3\") " pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.324446 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.324358    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhbbb"
Sep 11 19:34:19.334704 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:34:19.334679    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5392b55e_78e6_4494_8fbe_f5387f6c4cb3.slice/crio-a07a94be7b59ea5d3b821ff56667ed99ddc766d6ab4f05624d3477f015883379 WatchSource:0}: Error finding container a07a94be7b59ea5d3b821ff56667ed99ddc766d6ab4f05624d3477f015883379: Status 404 returned error can't find the container with id a07a94be7b59ea5d3b821ff56667ed99ddc766d6ab4f05624d3477f015883379
Sep 11 19:34:19.865994 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:19.865956    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhbbb" event={"ID":"5392b55e-78e6-4494-8fbe-f5387f6c4cb3","Type":"ContainerStarted","Data":"a07a94be7b59ea5d3b821ff56667ed99ddc766d6ab4f05624d3477f015883379"}
Sep 11 19:34:20.841211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:20.841175    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5wg6v"
Sep 11 19:34:20.873033 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:20.872998    2571 generic.go:356] "Generic (PLEG): container finished" podID="5392b55e-78e6-4494-8fbe-f5387f6c4cb3" containerID="f328381eebf969c195fa1f632482581ec342f5975009829f8f31cb11237af1e0" exitCode=0
Sep 11 19:34:20.873204 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:20.873081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhbbb" event={"ID":"5392b55e-78e6-4494-8fbe-f5387f6c4cb3","Type":"ContainerDied","Data":"f328381eebf969c195fa1f632482581ec342f5975009829f8f31cb11237af1e0"}
Sep 11 19:34:21.878133 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:21.878082    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhbbb" event={"ID":"5392b55e-78e6-4494-8fbe-f5387f6c4cb3","Type":"ContainerStarted","Data":"fd8c4bee219d2198836e44ae43dcdfe4098434ec128e79b42b8178d2d8db004b"}
Sep 11 19:34:21.878523 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:21.878155    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhbbb" event={"ID":"5392b55e-78e6-4494-8fbe-f5387f6c4cb3","Type":"ContainerStarted","Data":"eff63154ef6c9f12a8ba295c844dd049c3798e147e8d0ea755406f5fd40adf50"}
Sep 11 19:34:21.897799 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:21.897753    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mhbbb" podStartSLOduration=2.830563519 podStartE2EDuration="3.897737074s" totalImagesPullingTime="1.007314778s" totalInitContainerRuntime="59.858777ms" isStatefulPod=true podCreationTimestamp="2026-09-11 19:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:34:21.896493583 +0000 UTC m=+298.572430503" watchObservedRunningTime="2026-09-11 19:34:21.897737074 +0000 UTC m=+298.573673976"
Sep 11 19:34:22.867872 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:22.867843    2571 scope.go:117] "RemoveContainer" containerID="c42fff6bf31fcc41897dba9187573e967677539222fca9fe97c57db9061595a3"
Sep 11 19:34:23.688439 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:23.688410    2571 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 19:34:23.868894 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:23.868705    2571 scope.go:117] "RemoveContainer" containerID="5b3911dbc180448bd012be8a30028e41611e18668dc25de818216c91dd04ece1"
Sep 11 19:34:23.892868 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:23.891007    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7668d66d58-72spw" event={"ID":"6ae293b8-32ac-4b05-8dea-fe74ee042b07","Type":"ContainerStarted","Data":"d1da4ef1ad5b097ceeb073bb5a9aee2742c34c81b06208d870031f67166ebf3e"}
Sep 11 19:34:24.156717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:24.156684    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bf7847486-ccpsb"]
Sep 11 19:34:24.161027 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:24.161000    2571 patch_prober.go:28] interesting pod/image-registry-5bf7847486-ccpsb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 19:34:24.161204 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:24.161044    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" podUID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:34:24.896013 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:24.895984    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t" event={"ID":"0c7b79ab-d2aa-4fe5-887f-4f7968b27d4c","Type":"ContainerStarted","Data":"b1dcde6610af5527fedba125e0a365242701d4ce7fdef00cbcac4ff4a7166542"}
Sep 11 19:34:24.896417 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:24.896313    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:34:24.897827 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:24.897806    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c7f9fbff-rtl5t"
Sep 11 19:34:28.085853 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:28.085816    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" podUID="f2778046-0e46-405e-a116-bb31fa413068" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 19:34:29.867598 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:29.867556    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:34:34.161143 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:34.161098    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:34:38.085810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:38.085767    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" podUID="f2778046-0e46-405e-a116-bb31fa413068" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 19:34:48.085848 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:48.085799    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" podUID="f2778046-0e46-405e-a116-bb31fa413068" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 19:34:48.086231 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:48.085879    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6"
Sep 11 19:34:48.086412 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:48.086353    2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"26407a5286b98ab20564cf125903d4803b94e466ac2881086f7cc16ce86f5f45"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 11 19:34:48.086451 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:48.086434    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" podUID="f2778046-0e46-405e-a116-bb31fa413068" containerName="service-proxy" containerID="cri-o://26407a5286b98ab20564cf125903d4803b94e466ac2881086f7cc16ce86f5f45" gracePeriod=30
Sep 11 19:34:48.976286 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:48.976251    2571 generic.go:356] "Generic (PLEG): container finished" podID="f2778046-0e46-405e-a116-bb31fa413068" containerID="26407a5286b98ab20564cf125903d4803b94e466ac2881086f7cc16ce86f5f45" exitCode=2
Sep 11 19:34:48.976454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:48.976325    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" event={"ID":"f2778046-0e46-405e-a116-bb31fa413068","Type":"ContainerDied","Data":"26407a5286b98ab20564cf125903d4803b94e466ac2881086f7cc16ce86f5f45"}
Sep 11 19:34:48.976454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:48.976370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d7b8bcc4c-tt9h6" event={"ID":"f2778046-0e46-405e-a116-bb31fa413068","Type":"ContainerStarted","Data":"768dfb9e6fb317f9f3aa9f345bbc0efd33cedf155e0283d4bd80ab47fba88ef5"}
Sep 11 19:34:49.175281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.175241    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" podUID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" containerName="registry" containerID="cri-o://e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345" gracePeriod=30
Sep 11 19:34:49.415732 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.415706    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:34:49.452281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.452250    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkrj9\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-kube-api-access-vkrj9\") pod \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") "
Sep 11 19:34:49.452281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.452284    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-ca-trust-extracted\") pod \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") "
Sep 11 19:34:49.452501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.452309    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-installation-pull-secrets\") pod \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") "
Sep 11 19:34:49.452501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.452332    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") pod \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") "
Sep 11 19:34:49.452501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.452372    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-image-registry-private-configuration\") pod \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") "
Sep 11 19:34:49.452501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.452411    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-certificates\") pod \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") "
Sep 11 19:34:49.452501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.452454    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-bound-sa-token\") pod \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") "
Sep 11 19:34:49.452501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.452499    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-trusted-ca\") pod \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\" (UID: \"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50\") "
Sep 11 19:34:49.453138 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.453085    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:34:49.453367 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.453341    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:34:49.455460 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.455412    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:34:49.455460 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.455422    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-kube-api-access-vkrj9" (OuterVolumeSpecName: "kube-api-access-vkrj9") pod "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"). InnerVolumeSpecName "kube-api-access-vkrj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:34:49.455460 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.455430    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:34:49.455607 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.455493    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:34:49.455681 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.455660    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:34:49.461016 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.460993    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" (UID: "a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:34:49.553217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.553135    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-certificates\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:34:49.553217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.553163    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-bound-sa-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:34:49.553217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.553172    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-trusted-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:34:49.553217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.553182    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkrj9\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-kube-api-access-vkrj9\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:34:49.553217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.553192    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-ca-trust-extracted\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:34:49.553217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.553201    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-installation-pull-secrets\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:34:49.553217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.553208    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-registry-tls\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:34:49.553217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.553220    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50-image-registry-private-configuration\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:34:49.980811 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.980778    2571 generic.go:356] "Generic (PLEG): container finished" podID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" containerID="e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345" exitCode=0
Sep 11 19:34:49.980967 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.980817    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb"
Sep 11 19:34:49.980967 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.980823    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" event={"ID":"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50","Type":"ContainerDied","Data":"e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345"}
Sep 11 19:34:49.980967 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.980862    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf7847486-ccpsb" event={"ID":"a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50","Type":"ContainerDied","Data":"169a288fd6b17def3025d66845d5fb708f62b774c890fac2aa4d82003cb19f9a"}
Sep 11 19:34:49.980967 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.980880    2571 scope.go:117] "RemoveContainer" containerID="e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345"
Sep 11 19:34:49.989058 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.989040    2571 scope.go:117] "RemoveContainer" containerID="e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345"
Sep 11 19:34:49.989347 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:34:49.989324    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345\": container with ID starting with e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345 not found: ID does not exist" containerID="e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345"
Sep 11 19:34:49.989427 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.989357    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345"} err="failed to get container status \"e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345\": rpc error: code = NotFound desc = could not find container \"e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345\": container with ID starting with e6b1fe4d507cc6a06139dd042f53bb94882e8d1b34b8ed39304c6a1061a44345 not found: ID does not exist"
Sep 11 19:34:49.998835 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:49.998811    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bf7847486-ccpsb"]
Sep 11 19:34:50.002303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:50.002280    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bf7847486-ccpsb"]
Sep 11 19:34:51.875288 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:34:51.872905    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" path="/var/lib/kubelet/pods/a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50/volumes"
Sep 11 19:35:01.245478 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:01.245448    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tghl6_b5006eaf-06a5-4489-a487-826bf6640a1b/serve-healthcheck-canary/0.log"
Sep 11 19:35:37.678241 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:37.678198    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:35:37.680630 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:37.680604    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d2d85-34c6-44f9-a0e4-17ae596fea59-metrics-certs\") pod \"network-metrics-daemon-mtpxz\" (UID: \"d81d2d85-34c6-44f9-a0e4-17ae596fea59\") " pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:35:37.971841 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:37.971754    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7w7kq\""
Sep 11 19:35:37.979240 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:37.979216    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mtpxz"
Sep 11 19:35:38.249356 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:38.249269    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mtpxz"]
Sep 11 19:35:38.253048 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:35:38.253018    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81d2d85_34c6_44f9_a0e4_17ae596fea59.slice/crio-6005965d913d0902d59dc47c9f4af5f904763fc6bc553234987a2c37710237a8 WatchSource:0}: Error finding container 6005965d913d0902d59dc47c9f4af5f904763fc6bc553234987a2c37710237a8: Status 404 returned error can't find the container with id 6005965d913d0902d59dc47c9f4af5f904763fc6bc553234987a2c37710237a8
Sep 11 19:35:38.254980 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:38.254962    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:35:39.143665 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:39.143623    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtpxz" event={"ID":"d81d2d85-34c6-44f9-a0e4-17ae596fea59","Type":"ContainerStarted","Data":"6005965d913d0902d59dc47c9f4af5f904763fc6bc553234987a2c37710237a8"}
Sep 11 19:35:40.148602 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:40.148557    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtpxz" event={"ID":"d81d2d85-34c6-44f9-a0e4-17ae596fea59","Type":"ContainerStarted","Data":"e10eb5816bd1e43190d058441fa1e549493f47bf44c8158ae8965e6325b07206"}
Sep 11 19:35:40.149007 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:40.148625    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mtpxz" event={"ID":"d81d2d85-34c6-44f9-a0e4-17ae596fea59","Type":"ContainerStarted","Data":"38849c460c10f45e1e59219dc069b85170f56c063363bbfca5820ca5c2dcfcfb"}
Sep 11 19:35:40.170117 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:35:40.170057    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mtpxz" podStartSLOduration=375.215694631 podStartE2EDuration="6m16.170040813s" totalImagesPullingTime="954.346182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:40.169827449 +0000 UTC m=+376.845764357" watchObservedRunningTime="2026-09-11 19:35:40.170040813 +0000 UTC m=+376.845977721"
Sep 11 19:38:05.384484 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.384451    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"]
Sep 11 19:38:05.384926 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.384686    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" containerName="registry"
Sep 11 19:38:05.384926 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.384697    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" containerName="registry"
Sep 11 19:38:05.384926 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.384743    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10d1ce3-6a6a-44d3-bdd3-676adfa4ac50" containerName="registry"
Sep 11 19:38:05.386532 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.386515    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.391880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.391859    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 19:38:05.391986 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.391859    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z9lkf\""
Sep 11 19:38:05.393121 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.393089    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 19:38:05.404504 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.404482    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"]
Sep 11 19:38:05.433353 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.433328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlwm\" (UniqueName: \"kubernetes.io/projected/c1ddb702-5e4a-4328-b393-c9e48210bc1c-kube-api-access-4mlwm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.433448 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.433362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.433448 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.433381    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.533695 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.533667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.533789 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.533697    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.533789 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.533740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlwm\" (UniqueName: \"kubernetes.io/projected/c1ddb702-5e4a-4328-b393-c9e48210bc1c-kube-api-access-4mlwm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.534028 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.534011    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.534069 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.534037    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.544335 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.544308    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlwm\" (UniqueName: \"kubernetes.io/projected/c1ddb702-5e4a-4328-b393-c9e48210bc1c-kube-api-access-4mlwm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.695083 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.695022    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:05.961444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:05.961416    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"]
Sep 11 19:38:05.963544 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:38:05.963517    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ddb702_5e4a_4328_b393_c9e48210bc1c.slice/crio-8f7de9673dcdaa70b91d9edf0d0cc951188eddfe759d009e6c7f582ad529ee3a WatchSource:0}: Error finding container 8f7de9673dcdaa70b91d9edf0d0cc951188eddfe759d009e6c7f582ad529ee3a: Status 404 returned error can't find the container with id 8f7de9673dcdaa70b91d9edf0d0cc951188eddfe759d009e6c7f582ad529ee3a
Sep 11 19:38:06.607891 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:06.607856    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x" event={"ID":"c1ddb702-5e4a-4328-b393-c9e48210bc1c","Type":"ContainerStarted","Data":"8f7de9673dcdaa70b91d9edf0d0cc951188eddfe759d009e6c7f582ad529ee3a"}
Sep 11 19:38:11.627541 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:11.627506    2571 generic.go:356] "Generic (PLEG): container finished" podID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerID="22f3ebd44c5664ee27efa162c537db1ea20bf8752b99eeef5baa971ecaa85338" exitCode=0
Sep 11 19:38:11.627923 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:11.627589    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x" event={"ID":"c1ddb702-5e4a-4328-b393-c9e48210bc1c","Type":"ContainerDied","Data":"22f3ebd44c5664ee27efa162c537db1ea20bf8752b99eeef5baa971ecaa85338"}
Sep 11 19:38:13.640245 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:13.640211    2571 generic.go:356] "Generic (PLEG): container finished" podID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerID="b1b942cea07fe46693cd1e1a65c9df98e809b274b92445142d0fa07111df0284" exitCode=0
Sep 11 19:38:13.640637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:13.640298    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x" event={"ID":"c1ddb702-5e4a-4328-b393-c9e48210bc1c","Type":"ContainerDied","Data":"b1b942cea07fe46693cd1e1a65c9df98e809b274b92445142d0fa07111df0284"}
Sep 11 19:38:19.663862 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:19.663824    2571 generic.go:356] "Generic (PLEG): container finished" podID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerID="c7440a8294b6e175d3aa2f11e833b7884d923f355395fb29f0aeda6bed19450f" exitCode=0
Sep 11 19:38:19.664255 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:19.663892    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x" event={"ID":"c1ddb702-5e4a-4328-b393-c9e48210bc1c","Type":"ContainerDied","Data":"c7440a8294b6e175d3aa2f11e833b7884d923f355395fb29f0aeda6bed19450f"}
Sep 11 19:38:20.788127 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:20.788092    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:20.952551 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:20.952470    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-util\") pod \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") "
Sep 11 19:38:20.952551 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:20.952521    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mlwm\" (UniqueName: \"kubernetes.io/projected/c1ddb702-5e4a-4328-b393-c9e48210bc1c-kube-api-access-4mlwm\") pod \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") "
Sep 11 19:38:20.952720 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:20.952574    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-bundle\") pod \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\" (UID: \"c1ddb702-5e4a-4328-b393-c9e48210bc1c\") "
Sep 11 19:38:20.953440 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:20.953407    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-bundle" (OuterVolumeSpecName: "bundle") pod "c1ddb702-5e4a-4328-b393-c9e48210bc1c" (UID: "c1ddb702-5e4a-4328-b393-c9e48210bc1c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:38:20.954713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:20.954686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ddb702-5e4a-4328-b393-c9e48210bc1c-kube-api-access-4mlwm" (OuterVolumeSpecName: "kube-api-access-4mlwm") pod "c1ddb702-5e4a-4328-b393-c9e48210bc1c" (UID: "c1ddb702-5e4a-4328-b393-c9e48210bc1c"). InnerVolumeSpecName "kube-api-access-4mlwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:38:20.956765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:20.956746    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-util" (OuterVolumeSpecName: "util") pod "c1ddb702-5e4a-4328-b393-c9e48210bc1c" (UID: "c1ddb702-5e4a-4328-b393-c9e48210bc1c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:38:21.053701 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:21.053672    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-bundle\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:38:21.053701 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:21.053704    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1ddb702-5e4a-4328-b393-c9e48210bc1c-util\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:38:21.053891 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:21.053713    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mlwm\" (UniqueName: \"kubernetes.io/projected/c1ddb702-5e4a-4328-b393-c9e48210bc1c-kube-api-access-4mlwm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:38:21.672281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:21.672248    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x" event={"ID":"c1ddb702-5e4a-4328-b393-c9e48210bc1c","Type":"ContainerDied","Data":"8f7de9673dcdaa70b91d9edf0d0cc951188eddfe759d009e6c7f582ad529ee3a"}
Sep 11 19:38:21.672281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:21.672285    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407mq4x"
Sep 11 19:38:21.672559 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:21.672296    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7de9673dcdaa70b91d9edf0d0cc951188eddfe759d009e6c7f582ad529ee3a"
Sep 11 19:38:28.562006 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.561975    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"]
Sep 11 19:38:28.562366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.562189    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerName="extract"
Sep 11 19:38:28.562366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.562199    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerName="extract"
Sep 11 19:38:28.562366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.562216    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerName="pull"
Sep 11 19:38:28.562366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.562221    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerName="pull"
Sep 11 19:38:28.562366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.562231    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerName="util"
Sep 11 19:38:28.562366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.562237    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerName="util"
Sep 11 19:38:28.562366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.562288    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ddb702-5e4a-4328-b393-c9e48210bc1c" containerName="extract"
Sep 11 19:38:28.595780 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.595754    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"]
Sep 11 19:38:28.595915 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.595840    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"
Sep 11 19:38:28.598799 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.598777    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:38:28.599025 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.599009    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 19:38:28.599119 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.599014    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zl6zw\""
Sep 11 19:38:28.704245 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.704213    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182b46da-07e9-4f50-b706-062dd538b35e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfgx8\" (UID: \"182b46da-07e9-4f50-b706-062dd538b35e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"
Sep 11 19:38:28.704382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.704271    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpffv\" (UniqueName: \"kubernetes.io/projected/182b46da-07e9-4f50-b706-062dd538b35e-kube-api-access-kpffv\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfgx8\" (UID: \"182b46da-07e9-4f50-b706-062dd538b35e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"
Sep 11 19:38:28.805503 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.805475    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpffv\" (UniqueName: \"kubernetes.io/projected/182b46da-07e9-4f50-b706-062dd538b35e-kube-api-access-kpffv\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfgx8\" (UID: \"182b46da-07e9-4f50-b706-062dd538b35e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"
Sep 11 19:38:28.805588 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.805521    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182b46da-07e9-4f50-b706-062dd538b35e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfgx8\" (UID: \"182b46da-07e9-4f50-b706-062dd538b35e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"
Sep 11 19:38:28.805837 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.805822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182b46da-07e9-4f50-b706-062dd538b35e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfgx8\" (UID: \"182b46da-07e9-4f50-b706-062dd538b35e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"
Sep 11 19:38:28.817147 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.817063    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpffv\" (UniqueName: \"kubernetes.io/projected/182b46da-07e9-4f50-b706-062dd538b35e-kube-api-access-kpffv\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfgx8\" (UID: \"182b46da-07e9-4f50-b706-062dd538b35e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"
Sep 11 19:38:28.903837 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:28.903810    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"
Sep 11 19:38:29.197989 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:29.197957    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8"]
Sep 11 19:38:29.199783 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:38:29.199758    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182b46da_07e9_4f50_b706_062dd538b35e.slice/crio-73ab3d4153fb53bf561d0bd3ecdfb1d91f52da36d81596df72e0bd925e485ed7 WatchSource:0}: Error finding container 73ab3d4153fb53bf561d0bd3ecdfb1d91f52da36d81596df72e0bd925e485ed7: Status 404 returned error can't find the container with id 73ab3d4153fb53bf561d0bd3ecdfb1d91f52da36d81596df72e0bd925e485ed7
Sep 11 19:38:29.699968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:29.699936    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8" event={"ID":"182b46da-07e9-4f50-b706-062dd538b35e","Type":"ContainerStarted","Data":"73ab3d4153fb53bf561d0bd3ecdfb1d91f52da36d81596df72e0bd925e485ed7"}
Sep 11 19:38:44.754361 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:44.754321    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8" event={"ID":"182b46da-07e9-4f50-b706-062dd538b35e","Type":"ContainerStarted","Data":"601e5bfa0862d9763586780a9eb71584f2b84f2abc376a87e584ae3079a9376f"}
Sep 11 19:38:44.782833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:44.782779    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfgx8" podStartSLOduration=1.310543223 podStartE2EDuration="16.782761674s" totalImagesPullingTime="15.472218451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:44.776160878 +0000 UTC m=+561.452097788" watchObservedRunningTime="2026-09-11 19:38:44.782761674 +0000 UTC m=+561.458698582"
Sep 11 19:38:46.192115 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.192081    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"]
Sep 11 19:38:46.194186 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.194169    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.196673 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.196649    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z9lkf\""
Sep 11 19:38:46.196982 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.196967    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 19:38:46.197885 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.197867    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 19:38:46.204494 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.204474    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"]
Sep 11 19:38:46.315029 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.314996    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.315187 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.315040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.315187 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.315129    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f25m\" (UniqueName: \"kubernetes.io/projected/c58445b7-4788-4352-a8bb-8f7d7a608489-kube-api-access-6f25m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.416283 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.416257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f25m\" (UniqueName: \"kubernetes.io/projected/c58445b7-4788-4352-a8bb-8f7d7a608489-kube-api-access-6f25m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.416364 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.416310    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.416364 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.416346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.416779 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.416758    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.416840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.416760    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.425394 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.425373    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f25m\" (UniqueName: \"kubernetes.io/projected/c58445b7-4788-4352-a8bb-8f7d7a608489-kube-api-access-6f25m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.503252 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.503197    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:46.771494 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:46.771424    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"]
Sep 11 19:38:46.774126 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:38:46.774080    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58445b7_4788_4352_a8bb_8f7d7a608489.slice/crio-2b7ccfb39e76fe84a4b28d7837020c96aa61d91f6797a900f1da5b34eb4776cc WatchSource:0}: Error finding container 2b7ccfb39e76fe84a4b28d7837020c96aa61d91f6797a900f1da5b34eb4776cc: Status 404 returned error can't find the container with id 2b7ccfb39e76fe84a4b28d7837020c96aa61d91f6797a900f1da5b34eb4776cc
Sep 11 19:38:47.766073 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:47.765990    2571 generic.go:356] "Generic (PLEG): container finished" podID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerID="651139648fa4309bf2c5b0b452ebb166b9275a0e8d351045f07453476cf2a843" exitCode=0
Sep 11 19:38:47.766419 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:47.766072    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf" event={"ID":"c58445b7-4788-4352-a8bb-8f7d7a608489","Type":"ContainerDied","Data":"651139648fa4309bf2c5b0b452ebb166b9275a0e8d351045f07453476cf2a843"}
Sep 11 19:38:47.766419 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:47.766146    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf" event={"ID":"c58445b7-4788-4352-a8bb-8f7d7a608489","Type":"ContainerStarted","Data":"2b7ccfb39e76fe84a4b28d7837020c96aa61d91f6797a900f1da5b34eb4776cc"}
Sep 11 19:38:49.776232 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:49.776202    2571 generic.go:356] "Generic (PLEG): container finished" podID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerID="125d570a2102cc4ff6c5334cad2f3ef0fdc81f088b732692435cc5a11f4bde73" exitCode=0
Sep 11 19:38:49.776557 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:49.776266    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf" event={"ID":"c58445b7-4788-4352-a8bb-8f7d7a608489","Type":"ContainerDied","Data":"125d570a2102cc4ff6c5334cad2f3ef0fdc81f088b732692435cc5a11f4bde73"}
Sep 11 19:38:50.785587 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:50.782356    2571 generic.go:356] "Generic (PLEG): container finished" podID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerID="c7278507a3e9afbb9fbc19191d66999cec8333834bb4cac794105419d979e96d" exitCode=0
Sep 11 19:38:50.785587 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:50.782515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf" event={"ID":"c58445b7-4788-4352-a8bb-8f7d7a608489","Type":"ContainerDied","Data":"c7278507a3e9afbb9fbc19191d66999cec8333834bb4cac794105419d979e96d"}
Sep 11 19:38:51.901340 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:51.901317    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:52.058326 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.058261    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f25m\" (UniqueName: \"kubernetes.io/projected/c58445b7-4788-4352-a8bb-8f7d7a608489-kube-api-access-6f25m\") pod \"c58445b7-4788-4352-a8bb-8f7d7a608489\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") "
Sep 11 19:38:52.058326 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.058295    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-bundle\") pod \"c58445b7-4788-4352-a8bb-8f7d7a608489\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") "
Sep 11 19:38:52.058326 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.058321    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-util\") pod \"c58445b7-4788-4352-a8bb-8f7d7a608489\" (UID: \"c58445b7-4788-4352-a8bb-8f7d7a608489\") "
Sep 11 19:38:52.058724 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.058700    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-bundle" (OuterVolumeSpecName: "bundle") pod "c58445b7-4788-4352-a8bb-8f7d7a608489" (UID: "c58445b7-4788-4352-a8bb-8f7d7a608489"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:38:52.060356 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.060332    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58445b7-4788-4352-a8bb-8f7d7a608489-kube-api-access-6f25m" (OuterVolumeSpecName: "kube-api-access-6f25m") pod "c58445b7-4788-4352-a8bb-8f7d7a608489" (UID: "c58445b7-4788-4352-a8bb-8f7d7a608489"). InnerVolumeSpecName "kube-api-access-6f25m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:38:52.063885 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.063848    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-util" (OuterVolumeSpecName: "util") pod "c58445b7-4788-4352-a8bb-8f7d7a608489" (UID: "c58445b7-4788-4352-a8bb-8f7d7a608489"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:38:52.159020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.158990    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f25m\" (UniqueName: \"kubernetes.io/projected/c58445b7-4788-4352-a8bb-8f7d7a608489-kube-api-access-6f25m\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:38:52.159020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.159018    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-bundle\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:38:52.159190 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.159032    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c58445b7-4788-4352-a8bb-8f7d7a608489-util\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:38:52.791972 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.791936    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf" event={"ID":"c58445b7-4788-4352-a8bb-8f7d7a608489","Type":"ContainerDied","Data":"2b7ccfb39e76fe84a4b28d7837020c96aa61d91f6797a900f1da5b34eb4776cc"}
Sep 11 19:38:52.791972 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.791963    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv8dtf"
Sep 11 19:38:52.792195 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:52.791990    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b7ccfb39e76fe84a4b28d7837020c96aa61d91f6797a900f1da5b34eb4776cc"
Sep 11 19:38:55.185972 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.185940    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"]
Sep 11 19:38:55.186330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.186174    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerName="pull"
Sep 11 19:38:55.186330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.186186    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerName="pull"
Sep 11 19:38:55.186330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.186195    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerName="util"
Sep 11 19:38:55.186330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.186201    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerName="util"
Sep 11 19:38:55.186330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.186211    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerName="extract"
Sep 11 19:38:55.186330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.186217    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerName="extract"
Sep 11 19:38:55.186330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.186257    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c58445b7-4788-4352-a8bb-8f7d7a608489" containerName="extract"
Sep 11 19:38:55.187907 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.187882    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"
Sep 11 19:38:55.190597 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.190579    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 19:38:55.191704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.191689    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6v885\""
Sep 11 19:38:55.191704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.191695    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:38:55.198458 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.198434    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"]
Sep 11 19:38:55.277596 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.277568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cea7bf3-322d-4ca8-9cf7-50654724afb6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwqf4\" (UID: \"1cea7bf3-322d-4ca8-9cf7-50654724afb6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"
Sep 11 19:38:55.277732 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.277603    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpgzq\" (UniqueName: \"kubernetes.io/projected/1cea7bf3-322d-4ca8-9cf7-50654724afb6-kube-api-access-rpgzq\") pod \"openshift-lws-operator-fd8ccff4c-hwqf4\" (UID: \"1cea7bf3-322d-4ca8-9cf7-50654724afb6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"
Sep 11 19:38:55.378620 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.378591    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpgzq\" (UniqueName: \"kubernetes.io/projected/1cea7bf3-322d-4ca8-9cf7-50654724afb6-kube-api-access-rpgzq\") pod \"openshift-lws-operator-fd8ccff4c-hwqf4\" (UID: \"1cea7bf3-322d-4ca8-9cf7-50654724afb6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"
Sep 11 19:38:55.378769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.378745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cea7bf3-322d-4ca8-9cf7-50654724afb6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwqf4\" (UID: \"1cea7bf3-322d-4ca8-9cf7-50654724afb6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"
Sep 11 19:38:55.379056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.379041    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cea7bf3-322d-4ca8-9cf7-50654724afb6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwqf4\" (UID: \"1cea7bf3-322d-4ca8-9cf7-50654724afb6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"
Sep 11 19:38:55.386525 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.386505    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpgzq\" (UniqueName: \"kubernetes.io/projected/1cea7bf3-322d-4ca8-9cf7-50654724afb6-kube-api-access-rpgzq\") pod \"openshift-lws-operator-fd8ccff4c-hwqf4\" (UID: \"1cea7bf3-322d-4ca8-9cf7-50654724afb6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"
Sep 11 19:38:55.496199 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.496095    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"
Sep 11 19:38:55.765397 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.765368    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4"]
Sep 11 19:38:55.767769 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:38:55.767743    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cea7bf3_322d_4ca8_9cf7_50654724afb6.slice/crio-fd41285d86c506575493095bf1437ac3e986561f717a23f0c2848805bd71d294 WatchSource:0}: Error finding container fd41285d86c506575493095bf1437ac3e986561f717a23f0c2848805bd71d294: Status 404 returned error can't find the container with id fd41285d86c506575493095bf1437ac3e986561f717a23f0c2848805bd71d294
Sep 11 19:38:55.804041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:55.804017    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4" event={"ID":"1cea7bf3-322d-4ca8-9cf7-50654724afb6","Type":"ContainerStarted","Data":"fd41285d86c506575493095bf1437ac3e986561f717a23f0c2848805bd71d294"}
Sep 11 19:38:57.812966 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:57.812932    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4" event={"ID":"1cea7bf3-322d-4ca8-9cf7-50654724afb6","Type":"ContainerStarted","Data":"944d632830b6d5899ce6ee512bbb3fba1d329daa51aed5a59604edd7cf04630f"}
Sep 11 19:38:57.829578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:38:57.829520    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwqf4" podStartSLOduration=1.473555428 podStartE2EDuration="2.829496257s" totalImagesPullingTime="1.355940829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:57.829297582 +0000 UTC m=+574.505234492" watchObservedRunningTime="2026-09-11 19:38:57.829496257 +0000 UTC m=+574.505433156"
Sep 11 19:39:00.596957 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.596920    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"]
Sep 11 19:39:00.599155 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.599139    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.601708 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.601685    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 19:39:00.602920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.602904    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 19:39:00.603027 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.602972    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z9lkf\""
Sep 11 19:39:00.608298 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.608275    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"]
Sep 11 19:39:00.720842 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.717488    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5gb\" (UniqueName: \"kubernetes.io/projected/022b9d85-997d-422b-819a-c389bf659d5f-kube-api-access-2v5gb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.720842 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.717553    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.720842 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.717608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.818746 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.818713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5gb\" (UniqueName: \"kubernetes.io/projected/022b9d85-997d-422b-819a-c389bf659d5f-kube-api-access-2v5gb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.818882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.818757    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.818882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.818794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.819331 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.819307    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.819399 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.819307    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.827265 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.827243    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5gb\" (UniqueName: \"kubernetes.io/projected/022b9d85-997d-422b-819a-c389bf659d5f-kube-api-access-2v5gb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:00.908190 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:00.908166    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:01.173172 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:01.173038    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"]
Sep 11 19:39:01.175443 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:39:01.175411    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022b9d85_997d_422b_819a_c389bf659d5f.slice/crio-6df3e528143657b9887bf9201cc50b0460aa1b487edc58380c209fec39e408ae WatchSource:0}: Error finding container 6df3e528143657b9887bf9201cc50b0460aa1b487edc58380c209fec39e408ae: Status 404 returned error can't find the container with id 6df3e528143657b9887bf9201cc50b0460aa1b487edc58380c209fec39e408ae
Sep 11 19:39:01.829822 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:01.829780    2571 generic.go:356] "Generic (PLEG): container finished" podID="022b9d85-997d-422b-819a-c389bf659d5f" containerID="04a389b1881392ee985622299cdc173ef054353f919dfd3cef84d87ef4ea92a4" exitCode=0
Sep 11 19:39:01.830167 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:01.829863    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94" event={"ID":"022b9d85-997d-422b-819a-c389bf659d5f","Type":"ContainerDied","Data":"04a389b1881392ee985622299cdc173ef054353f919dfd3cef84d87ef4ea92a4"}
Sep 11 19:39:01.830167 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:01.829915    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94" event={"ID":"022b9d85-997d-422b-819a-c389bf659d5f","Type":"ContainerStarted","Data":"6df3e528143657b9887bf9201cc50b0460aa1b487edc58380c209fec39e408ae"}
Sep 11 19:39:02.835401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:02.835302    2571 generic.go:356] "Generic (PLEG): container finished" podID="022b9d85-997d-422b-819a-c389bf659d5f" containerID="f79090701ba5b879641b0df813ed2dc939fc305661a1a0e2bdb10b3b7531ce4f" exitCode=0
Sep 11 19:39:02.835401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:02.835389    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94" event={"ID":"022b9d85-997d-422b-819a-c389bf659d5f","Type":"ContainerDied","Data":"f79090701ba5b879641b0df813ed2dc939fc305661a1a0e2bdb10b3b7531ce4f"}
Sep 11 19:39:03.841866 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:03.841829    2571 generic.go:356] "Generic (PLEG): container finished" podID="022b9d85-997d-422b-819a-c389bf659d5f" containerID="de1bd45fdcb5df54ed7686d9be6e3c39f164e0cd1123f167427d7eaf5dfbbfa1" exitCode=0
Sep 11 19:39:03.842255 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:03.841917    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94" event={"ID":"022b9d85-997d-422b-819a-c389bf659d5f","Type":"ContainerDied","Data":"de1bd45fdcb5df54ed7686d9be6e3c39f164e0cd1123f167427d7eaf5dfbbfa1"}
Sep 11 19:39:04.962642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:04.962621    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:05.147002 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.146980    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-util\") pod \"022b9d85-997d-422b-819a-c389bf659d5f\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") "
Sep 11 19:39:05.147168 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.147025    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v5gb\" (UniqueName: \"kubernetes.io/projected/022b9d85-997d-422b-819a-c389bf659d5f-kube-api-access-2v5gb\") pod \"022b9d85-997d-422b-819a-c389bf659d5f\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") "
Sep 11 19:39:05.147168 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.147080    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-bundle\") pod \"022b9d85-997d-422b-819a-c389bf659d5f\" (UID: \"022b9d85-997d-422b-819a-c389bf659d5f\") "
Sep 11 19:39:05.147789 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.147759    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-bundle" (OuterVolumeSpecName: "bundle") pod "022b9d85-997d-422b-819a-c389bf659d5f" (UID: "022b9d85-997d-422b-819a-c389bf659d5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:39:05.149040 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.149012    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022b9d85-997d-422b-819a-c389bf659d5f-kube-api-access-2v5gb" (OuterVolumeSpecName: "kube-api-access-2v5gb") pod "022b9d85-997d-422b-819a-c389bf659d5f" (UID: "022b9d85-997d-422b-819a-c389bf659d5f"). InnerVolumeSpecName "kube-api-access-2v5gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:39:05.155163 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.155119    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-util" (OuterVolumeSpecName: "util") pod "022b9d85-997d-422b-819a-c389bf659d5f" (UID: "022b9d85-997d-422b-819a-c389bf659d5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:39:05.248005 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.247985    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-bundle\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:39:05.248005 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.248003    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022b9d85-997d-422b-819a-c389bf659d5f-util\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:39:05.248149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.248012    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v5gb\" (UniqueName: \"kubernetes.io/projected/022b9d85-997d-422b-819a-c389bf659d5f-kube-api-access-2v5gb\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:39:05.851760 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.851725    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94" event={"ID":"022b9d85-997d-422b-819a-c389bf659d5f","Type":"ContainerDied","Data":"6df3e528143657b9887bf9201cc50b0460aa1b487edc58380c209fec39e408ae"}
Sep 11 19:39:05.851950 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.851777    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df3e528143657b9887bf9201cc50b0460aa1b487edc58380c209fec39e408ae"
Sep 11 19:39:05.851950 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:05.851743    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftmf94"
Sep 11 19:39:16.895634 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.895596    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"]
Sep 11 19:39:16.896085 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.895827    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022b9d85-997d-422b-819a-c389bf659d5f" containerName="extract"
Sep 11 19:39:16.896085 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.895836    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="022b9d85-997d-422b-819a-c389bf659d5f" containerName="extract"
Sep 11 19:39:16.896085 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.895849    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022b9d85-997d-422b-819a-c389bf659d5f" containerName="pull"
Sep 11 19:39:16.896085 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.895855    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="022b9d85-997d-422b-819a-c389bf659d5f" containerName="pull"
Sep 11 19:39:16.896085 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.895868    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022b9d85-997d-422b-819a-c389bf659d5f" containerName="util"
Sep 11 19:39:16.896085 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.895874    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="022b9d85-997d-422b-819a-c389bf659d5f" containerName="util"
Sep 11 19:39:16.896085 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.895912    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="022b9d85-997d-422b-819a-c389bf659d5f" containerName="extract"
Sep 11 19:39:16.898293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.898276    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:16.900834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.900814    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 19:39:16.902062 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.902047    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 19:39:16.902139 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.902047    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z9lkf\""
Sep 11 19:39:16.910305 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:16.910281    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"]
Sep 11 19:39:17.031685 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.031647    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.031878 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.031790    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdb9p\" (UniqueName: \"kubernetes.io/projected/863d208c-7754-4ede-a38f-0e0310e1ae7e-kube-api-access-tdb9p\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.031878 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.031865    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.132365 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.132329    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.132533 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.132392    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdb9p\" (UniqueName: \"kubernetes.io/projected/863d208c-7754-4ede-a38f-0e0310e1ae7e-kube-api-access-tdb9p\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.132533 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.132422    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.132786 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.132761    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.132821 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.132784    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.148725 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.148665    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdb9p\" (UniqueName: \"kubernetes.io/projected/863d208c-7754-4ede-a38f-0e0310e1ae7e-kube-api-access-tdb9p\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:17.206847 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:17.206815    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:19.292718 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.292687    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"]
Sep 11 19:39:19.294815 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.294799    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.298454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.298430    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 19:39:19.298578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.298526    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 19:39:19.300167 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.300145    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 19:39:19.300302 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.300145    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 19:39:19.300452 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.300435    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b7k7l\""
Sep 11 19:39:19.316286 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.316259    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"]
Sep 11 19:39:19.449624 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.449585    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.449797 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.449640    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.449797 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.449708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-tmp\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.449797 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.449777    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-manifests\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.449901 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.449813    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndwx\" (UniqueName: \"kubernetes.io/projected/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-kube-api-access-lndwx\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.550579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.550485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-tmp\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.550579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.550528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-manifests\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.550579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.550552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndwx\" (UniqueName: \"kubernetes.io/projected/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-kube-api-access-lndwx\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.550852 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.550662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.550852 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.550739    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.550957 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.550865    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-manifests\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.552802 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.552770    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-tmp\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.553154 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.553133    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.553203 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.553133    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-webhook-cert\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.561499 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.561478    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndwx\" (UniqueName: \"kubernetes.io/projected/bf216ac5-0777-4aaf-a0d5-ddca19b337a7-kube-api-access-lndwx\") pod \"opendatahub-operator-controller-manager-cccd5967-bp6t4\" (UID: \"bf216ac5-0777-4aaf-a0d5-ddca19b337a7\") " pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.605379 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.605344    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:19.793161 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:19.793129    2571 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 11 19:39:19.793161 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28" Netns:"/var/run/netns/4ce05568-0a10-4d8e-9e39-a38a775e01d4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:19.793161 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:19.793161 ip-10-0-136-27 kubenswrapper[2571]:  >
Sep 11 19:39:19.793324 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:19.793190    2571 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 11 19:39:19.793324 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28" Netns:"/var/run/netns/4ce05568-0a10-4d8e-9e39-a38a775e01d4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:19.793324 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:19.793324 ip-10-0-136-27 kubenswrapper[2571]:  > pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:19.793324 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:19.793207    2571 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Sep 11 19:39:19.793324 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28" Netns:"/var/run/netns/4ce05568-0a10-4d8e-9e39-a38a775e01d4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:19.793324 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:19.793324 ip-10-0-136-27 kubenswrapper[2571]:  > pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:19.793543 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:19.793268    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace(863d208c-7754-4ede-a38f-0e0310e1ae7e)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace(863d208c-7754-4ede-a38f-0e0310e1ae7e)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28\\\" Netns:\\\"/var/run/netns/4ce05568-0a10-4d8e-9e39-a38a775e01d4\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=ae2a676bb73afde534777631c74098bab1248abf1c4b3c973c294192f4258a28;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get \\\"https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e"
Sep 11 19:39:19.904797 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.904771    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:19.905045 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:19.905031    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:22.190807 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:22.190777    2571 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 11 19:39:22.190807 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d" Netns:"/var/run/netns/554e6a5d-ca4f-46cc-9999-2bee6d5405b6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:22.190807 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:22.190807 ip-10-0-136-27 kubenswrapper[2571]:  >
Sep 11 19:39:22.191210 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:22.190826    2571 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 11 19:39:22.191210 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d" Netns:"/var/run/netns/554e6a5d-ca4f-46cc-9999-2bee6d5405b6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:22.191210 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:22.191210 ip-10-0-136-27 kubenswrapper[2571]:  > pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:22.191210 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:22.190843    2571 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Sep 11 19:39:22.191210 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d" Netns:"/var/run/netns/554e6a5d-ca4f-46cc-9999-2bee6d5405b6" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:22.191210 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:22.191210 ip-10-0-136-27 kubenswrapper[2571]:  > pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:22.191613 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:22.190922    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub(bf216ac5-0777-4aaf-a0d5-ddca19b337a7)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub(bf216ac5-0777-4aaf-a0d5-ddca19b337a7)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d\\\" Netns:\\\"/var/run/netns/554e6a5d-ca4f-46cc-9999-2bee6d5405b6\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=7d23b442040a2d53d919372c285ca7d35a9b4e22e19b8e94eec801726c061e5d;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7\\\" Path:\\\"\\\" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get \\\"https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4" podUID="bf216ac5-0777-4aaf-a0d5-ddca19b337a7"
Sep 11 19:39:22.489742 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:22.489703    2571 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 11 19:39:22.489742 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7" Netns:"/var/run/netns/3ed304d6-4949-4b99-b750-a819d23c6042" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:22.489742 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:22.489742 ip-10-0-136-27 kubenswrapper[2571]:  >
Sep 11 19:39:22.489896 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:22.489765    2571 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 11 19:39:22.489896 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7" Netns:"/var/run/netns/3ed304d6-4949-4b99-b750-a819d23c6042" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:22.489896 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:22.489896 ip-10-0-136-27 kubenswrapper[2571]:  > pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:22.489896 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:22.489785    2571 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Sep 11 19:39:22.489896 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7" Netns:"/var/run/netns/3ed304d6-4949-4b99-b750-a819d23c6042" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:22.489896 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:22.489896 ip-10-0-136-27 kubenswrapper[2571]:  > pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:22.490137 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:22.489877    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace(863d208c-7754-4ede-a38f-0e0310e1ae7e)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace(863d208c-7754-4ede-a38f-0e0310e1ae7e)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7\\\" Netns:\\\"/var/run/netns/3ed304d6-4949-4b99-b750-a819d23c6042\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=8fcd46879a2217e5883aa4fa5560e8a49e6170e283ef771032361b0ae3f324e7;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get \\\"https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e"
Sep 11 19:39:22.916589 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:22.916559    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:22.916850 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:22.916838    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:25.502945 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:25.502904    2571 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 11 19:39:25.502945 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a" Netns:"/var/run/netns/feca7251-85c6-409f-9b1c-f19ce3d8d231" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:25.502945 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:25.502945 ip-10-0-136-27 kubenswrapper[2571]:  >
Sep 11 19:39:25.503338 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:25.502970    2571 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 11 19:39:25.503338 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a" Netns:"/var/run/netns/feca7251-85c6-409f-9b1c-f19ce3d8d231" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:25.503338 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:25.503338 ip-10-0-136-27 kubenswrapper[2571]:  > pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:25.503338 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:25.502990    2571 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Sep 11 19:39:25.503338 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a" Netns:"/var/run/netns/feca7251-85c6-409f-9b1c-f19ce3d8d231" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:25.503338 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:25.503338 ip-10-0-136-27 kubenswrapper[2571]:  > pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:25.503576 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:25.503055    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub(bf216ac5-0777-4aaf-a0d5-ddca19b337a7)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub(bf216ac5-0777-4aaf-a0d5-ddca19b337a7)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a\\\" Netns:\\\"/var/run/netns/feca7251-85c6-409f-9b1c-f19ce3d8d231\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=0b57379301f41c45703f2096f0bdc786f293b4e77bc2af5ba8b44d2f6f92027a;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7\\\" Path:\\\"\\\" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get \\\"https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4" podUID="bf216ac5-0777-4aaf-a0d5-ddca19b337a7"
Sep 11 19:39:35.867246 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:35.867212    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:35.867598 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:35.867530    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:37.867606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:37.867567    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:37.867974 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:37.867911    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:38.452918 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:38.452874    2571 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 11 19:39:38.452918 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e" Netns:"/var/run/netns/c178dc57-ffa0-400f-a07c-8d9d7ee33cb1" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:38.452918 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:38.452918 ip-10-0-136-27 kubenswrapper[2571]:  >
Sep 11 19:39:38.453164 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:38.452965    2571 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 11 19:39:38.453164 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e" Netns:"/var/run/netns/c178dc57-ffa0-400f-a07c-8d9d7ee33cb1" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:38.453164 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:38.453164 ip-10-0-136-27 kubenswrapper[2571]:  > pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:38.453164 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:38.452988    2571 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Sep 11 19:39:38.453164 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e" Netns:"/var/run/netns/c178dc57-ffa0-400f-a07c-8d9d7ee33cb1" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e" Path:"" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:38.453164 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:38.453164 ip-10-0-136-27 kubenswrapper[2571]:  > pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:38.453420 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:38.453086    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace(863d208c-7754-4ede-a38f-0e0310e1ae7e)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace(863d208c-7754-4ede-a38f-0e0310e1ae7e)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4_openshift-marketplace_863d208c-7754-4ede-a38f-0e0310e1ae7e_0(ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e): error adding pod openshift-marketplace_8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e\\\" Netns:\\\"/var/run/netns/c178dc57-ffa0-400f-a07c-8d9d7ee33cb1\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-marketplace;K8S_POD_NAME=8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4;K8S_POD_INFRA_CONTAINER_ID=ee8c0c29f17ca66537a70d20e1e592cc6cc39b21761f5fe55061f544b481b30e;K8S_POD_UID=863d208c-7754-4ede-a38f-0e0310e1ae7e\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4] networking: Multus: [openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4/863d208c-7754-4ede-a38f-0e0310e1ae7e]: error waiting for pod: Get \\\"https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-marketplace/pods/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e"
Sep 11 19:39:40.453782 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:40.453753    2571 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 11 19:39:40.453782 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd" Netns:"/var/run/netns/18742001-0398-447c-b016-e4662e31e08d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:40.453782 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:40.453782 ip-10-0-136-27 kubenswrapper[2571]:  >
Sep 11 19:39:40.454171 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:40.453802    2571 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 11 19:39:40.454171 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd" Netns:"/var/run/netns/18742001-0398-447c-b016-e4662e31e08d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:40.454171 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:40.454171 ip-10-0-136-27 kubenswrapper[2571]:  > pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:40.454171 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:40.453818    2571 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Sep 11 19:39:40.454171 ip-10-0-136-27 kubenswrapper[2571]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd" Netns:"/var/run/netns/18742001-0398-447c-b016-e4662e31e08d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7" Path:"" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get "https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s": context deadline exceeded
Sep 11 19:39:40.454171 ip-10-0-136-27 kubenswrapper[2571]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 11 19:39:40.454171 ip-10-0-136-27 kubenswrapper[2571]:  > pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:40.454403 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:39:40.453886    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub(bf216ac5-0777-4aaf-a0d5-ddca19b337a7)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub(bf216ac5-0777-4aaf-a0d5-ddca19b337a7)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_opendatahub-operator-controller-manager-cccd5967-bp6t4_opendatahub_bf216ac5-0777-4aaf-a0d5-ddca19b337a7_0(e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd): error adding pod opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd\\\" Netns:\\\"/var/run/netns/18742001-0398-447c-b016-e4662e31e08d\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=opendatahub;K8S_POD_NAME=opendatahub-operator-controller-manager-cccd5967-bp6t4;K8S_POD_INFRA_CONTAINER_ID=e94fd976093fc602c1421d6f5f1584e36e9225c9ddb77d764e9299d796bbfcdd;K8S_POD_UID=bf216ac5-0777-4aaf-a0d5-ddca19b337a7\\\" Path:\\\"\\\" ERRORED: error configuring pod [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4] networking: Multus: [opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4/bf216ac5-0777-4aaf-a0d5-ddca19b337a7]: error waiting for pod: Get \\\"https://af56d428111af48ccac855fff79e718d-863c02c859935246.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/opendatahub/pods/opendatahub-operator-controller-manager-cccd5967-bp6t4?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4" podUID="bf216ac5-0777-4aaf-a0d5-ddca19b337a7"
Sep 11 19:39:49.867746 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:49.867658    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:49.868291 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:49.867986    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:50.136772 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:50.136745    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"]
Sep 11 19:39:50.139092 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:39:50.139063    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863d208c_7754_4ede_a38f_0e0310e1ae7e.slice/crio-9912409076dffa0ab5bfadf8c3bedaf448b2e492499e1f05dd7d7d0fbfbc1e3c WatchSource:0}: Error finding container 9912409076dffa0ab5bfadf8c3bedaf448b2e492499e1f05dd7d7d0fbfbc1e3c: Status 404 returned error can't find the container with id 9912409076dffa0ab5bfadf8c3bedaf448b2e492499e1f05dd7d7d0fbfbc1e3c
Sep 11 19:39:51.025062 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:51.025023    2571 generic.go:356] "Generic (PLEG): container finished" podID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerID="3d1911e18ac549e8a27f909a26c1d11bc78a5196d1d9ef02553dfdcb8804b4dc" exitCode=0
Sep 11 19:39:51.025062 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:51.025066    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4" event={"ID":"863d208c-7754-4ede-a38f-0e0310e1ae7e","Type":"ContainerDied","Data":"3d1911e18ac549e8a27f909a26c1d11bc78a5196d1d9ef02553dfdcb8804b4dc"}
Sep 11 19:39:51.025609 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:51.025129    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4" event={"ID":"863d208c-7754-4ede-a38f-0e0310e1ae7e","Type":"ContainerStarted","Data":"9912409076dffa0ab5bfadf8c3bedaf448b2e492499e1f05dd7d7d0fbfbc1e3c"}
Sep 11 19:39:51.867592 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:51.867558    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:51.867834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:51.867819    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:52.037934 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:52.037898    2571 generic.go:356] "Generic (PLEG): container finished" podID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerID="6de2c797ddc456dc26885b83e737972e412b5f357b5bceb4ef782e9d8da497eb" exitCode=0
Sep 11 19:39:52.038312 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:52.037956    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4" event={"ID":"863d208c-7754-4ede-a38f-0e0310e1ae7e","Type":"ContainerDied","Data":"6de2c797ddc456dc26885b83e737972e412b5f357b5bceb4ef782e9d8da497eb"}
Sep 11 19:39:52.143687 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:52.143659    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"]
Sep 11 19:39:52.144299 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:39:52.144272    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf216ac5_0777_4aaf_a0d5_ddca19b337a7.slice/crio-e8ebc55bf8eb6b8433329cd25ec3b027e76cb6f5ee5c3b58687dc81d28f6bbe3 WatchSource:0}: Error finding container e8ebc55bf8eb6b8433329cd25ec3b027e76cb6f5ee5c3b58687dc81d28f6bbe3: Status 404 returned error can't find the container with id e8ebc55bf8eb6b8433329cd25ec3b027e76cb6f5ee5c3b58687dc81d28f6bbe3
Sep 11 19:39:53.046728 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:53.046656    2571 generic.go:356] "Generic (PLEG): container finished" podID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerID="cdbe76b54e58ada8ec787c96d365f5efff21e6135c7a68bdb3f0bf950f58ab03" exitCode=0
Sep 11 19:39:53.047215 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:53.046739    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4" event={"ID":"863d208c-7754-4ede-a38f-0e0310e1ae7e","Type":"ContainerDied","Data":"cdbe76b54e58ada8ec787c96d365f5efff21e6135c7a68bdb3f0bf950f58ab03"}
Sep 11 19:39:53.048810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:53.048770    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4" event={"ID":"bf216ac5-0777-4aaf-a0d5-ddca19b337a7","Type":"ContainerStarted","Data":"e8ebc55bf8eb6b8433329cd25ec3b027e76cb6f5ee5c3b58687dc81d28f6bbe3"}
Sep 11 19:39:54.899170 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:54.899146    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:54.998796 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:54.998775    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-bundle\") pod \"863d208c-7754-4ede-a38f-0e0310e1ae7e\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") "
Sep 11 19:39:54.998889 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:54.998843    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdb9p\" (UniqueName: \"kubernetes.io/projected/863d208c-7754-4ede-a38f-0e0310e1ae7e-kube-api-access-tdb9p\") pod \"863d208c-7754-4ede-a38f-0e0310e1ae7e\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") "
Sep 11 19:39:54.998889 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:54.998882    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-util\") pod \"863d208c-7754-4ede-a38f-0e0310e1ae7e\" (UID: \"863d208c-7754-4ede-a38f-0e0310e1ae7e\") "
Sep 11 19:39:54.999639 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:54.999609    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-bundle" (OuterVolumeSpecName: "bundle") pod "863d208c-7754-4ede-a38f-0e0310e1ae7e" (UID: "863d208c-7754-4ede-a38f-0e0310e1ae7e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:39:55.000720 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.000690    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863d208c-7754-4ede-a38f-0e0310e1ae7e-kube-api-access-tdb9p" (OuterVolumeSpecName: "kube-api-access-tdb9p") pod "863d208c-7754-4ede-a38f-0e0310e1ae7e" (UID: "863d208c-7754-4ede-a38f-0e0310e1ae7e"). InnerVolumeSpecName "kube-api-access-tdb9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:39:55.004507 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.004365    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-util" (OuterVolumeSpecName: "util") pod "863d208c-7754-4ede-a38f-0e0310e1ae7e" (UID: "863d208c-7754-4ede-a38f-0e0310e1ae7e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:39:55.061575 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.061541    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4" event={"ID":"863d208c-7754-4ede-a38f-0e0310e1ae7e","Type":"ContainerDied","Data":"9912409076dffa0ab5bfadf8c3bedaf448b2e492499e1f05dd7d7d0fbfbc1e3c"}
Sep 11 19:39:55.061702 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.061597    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9912409076dffa0ab5bfadf8c3bedaf448b2e492499e1f05dd7d7d0fbfbc1e3c"
Sep 11 19:39:55.061702 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.061552    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09d2k4"
Sep 11 19:39:55.063231 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.063199    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4" event={"ID":"bf216ac5-0777-4aaf-a0d5-ddca19b337a7","Type":"ContainerStarted","Data":"860a825493b9803c0f3b26bc1c7ccc91f993061f42bdc2fa11ac42f29dacaa22"}
Sep 11 19:39:55.100328 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.100299    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-util\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:39:55.100440 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.100394    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/863d208c-7754-4ede-a38f-0e0310e1ae7e-bundle\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:39:55.100507 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:55.100486    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdb9p\" (UniqueName: \"kubernetes.io/projected/863d208c-7754-4ede-a38f-0e0310e1ae7e-kube-api-access-tdb9p\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:39:56.068898 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:56.068861    2571 generic.go:356] "Generic (PLEG): container finished" podID="bf216ac5-0777-4aaf-a0d5-ddca19b337a7" containerID="860a825493b9803c0f3b26bc1c7ccc91f993061f42bdc2fa11ac42f29dacaa22" exitCode=0
Sep 11 19:39:56.069282 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:56.068915    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4" event={"ID":"bf216ac5-0777-4aaf-a0d5-ddca19b337a7","Type":"ContainerDied","Data":"860a825493b9803c0f3b26bc1c7ccc91f993061f42bdc2fa11ac42f29dacaa22"}
Sep 11 19:39:57.074707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:57.074674    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4" event={"ID":"bf216ac5-0777-4aaf-a0d5-ddca19b337a7","Type":"ContainerStarted","Data":"9cf9cb39dab292e0cca35a19b66f85387cf849ca55b70d78f3ccabe08b117f06"}
Sep 11 19:39:57.075059 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:57.074761    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:39:57.096701 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:39:57.096662    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4" podStartSLOduration=33.978236921 podStartE2EDuration="38.09664972s" totalImagesPullingTime="3.10843515s" totalInitContainerRuntime="1.009977649s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:39:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:39:57.095908146 +0000 UTC m=+633.771845054" watchObservedRunningTime="2026-09-11 19:39:57.09664972 +0000 UTC m=+633.772586629"
Sep 11 19:40:08.081578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:08.081551    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cccd5967-bp6t4"
Sep 11 19:40:10.034420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.032127    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"]
Sep 11 19:40:10.034420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.032652    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerName="util"
Sep 11 19:40:10.034420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.032668    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerName="util"
Sep 11 19:40:10.034420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.032680    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerName="pull"
Sep 11 19:40:10.034420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.032688    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerName="pull"
Sep 11 19:40:10.034420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.032714    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerName="extract"
Sep 11 19:40:10.034420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.032723    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerName="extract"
Sep 11 19:40:10.034420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.032855    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="863d208c-7754-4ede-a38f-0e0310e1ae7e" containerName="extract"
Sep 11 19:40:10.042250 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.042229    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.045934 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.045910    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 19:40:10.046051 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.045935    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 19:40:10.046759 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.046745    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 19:40:10.046759 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.046752    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-thsxq\""
Sep 11 19:40:10.046864 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.046769    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 19:40:10.053954 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.053925    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"]
Sep 11 19:40:10.093484 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.093459    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tls-certs\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.093586 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.093510    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgg7s\" (UniqueName: \"kubernetes.io/projected/f176b659-74c0-4e11-8d2b-2b78a8cf663b-kube-api-access-kgg7s\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.093586 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.093534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tmp\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.194591 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.194560    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgg7s\" (UniqueName: \"kubernetes.io/projected/f176b659-74c0-4e11-8d2b-2b78a8cf663b-kube-api-access-kgg7s\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.194873 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.194601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tmp\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.194873 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.194636    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tls-certs\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.194873 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:40:10.194749    2571 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 19:40:10.194873 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:40:10.194828    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tls-certs podName:f176b659-74c0-4e11-8d2b-2b78a8cf663b nodeName:}" failed. No retries permitted until 2026-09-11 19:40:10.694807948 +0000 UTC m=+647.370744838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tls-certs") pod "kube-auth-proxy-5d55d58b74-dpn26" (UID: "f176b659-74c0-4e11-8d2b-2b78a8cf663b") : secret "kube-auth-proxy-tls" not found
Sep 11 19:40:10.197000 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.196978    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tmp\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.210884 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.210857    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgg7s\" (UniqueName: \"kubernetes.io/projected/f176b659-74c0-4e11-8d2b-2b78a8cf663b-kube-api-access-kgg7s\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.699030 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.698998    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tls-certs\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.701425 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.701402    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f176b659-74c0-4e11-8d2b-2b78a8cf663b-tls-certs\") pod \"kube-auth-proxy-5d55d58b74-dpn26\" (UID: \"f176b659-74c0-4e11-8d2b-2b78a8cf663b\") " pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:10.952320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:10.952237    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"
Sep 11 19:40:11.233117 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:11.233075    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26"]
Sep 11 19:40:11.234434 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:40:11.234392    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf176b659_74c0_4e11_8d2b_2b78a8cf663b.slice/crio-07fdd27f6480f976b0a05518cb40b453ff1dbe64ab46068dcee7e457b5b10d93 WatchSource:0}: Error finding container 07fdd27f6480f976b0a05518cb40b453ff1dbe64ab46068dcee7e457b5b10d93: Status 404 returned error can't find the container with id 07fdd27f6480f976b0a05518cb40b453ff1dbe64ab46068dcee7e457b5b10d93
Sep 11 19:40:12.135590 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:12.135515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26" event={"ID":"f176b659-74c0-4e11-8d2b-2b78a8cf663b","Type":"ContainerStarted","Data":"07fdd27f6480f976b0a05518cb40b453ff1dbe64ab46068dcee7e457b5b10d93"}
Sep 11 19:40:14.146432 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.146394    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26" event={"ID":"f176b659-74c0-4e11-8d2b-2b78a8cf663b","Type":"ContainerStarted","Data":"2722312a9eff706b5c048e101d6557c1252b788fa524235c325ede00e408171d"}
Sep 11 19:40:14.163083 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.162982    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d55d58b74-dpn26" podStartSLOduration=1.494637713 podStartE2EDuration="4.162965595s" totalImagesPullingTime="2.668327882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:14.16208542 +0000 UTC m=+650.838022330" watchObservedRunningTime="2026-09-11 19:40:14.162965595 +0000 UTC m=+650.838902539"
Sep 11 19:40:14.467865 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.467786    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"]
Sep 11 19:40:14.470938 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.470922    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.476755 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.476724    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 19:40:14.477020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.476767    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4qj94\""
Sep 11 19:40:14.477578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.477562    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 19:40:14.477650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.477622    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 19:40:14.493792 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.493754    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"]
Sep 11 19:40:14.527298 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.527269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71654660-e657-4ba8-9131-0e6ddbfed47f-manager-config\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.527409 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.527331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71654660-e657-4ba8-9131-0e6ddbfed47f-metrics-cert\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.527409 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.527388    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71654660-e657-4ba8-9131-0e6ddbfed47f-cert\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.527490 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.527415    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppkf2\" (UniqueName: \"kubernetes.io/projected/71654660-e657-4ba8-9131-0e6ddbfed47f-kube-api-access-ppkf2\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.627852 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.627822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71654660-e657-4ba8-9131-0e6ddbfed47f-manager-config\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.628013 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.627873    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71654660-e657-4ba8-9131-0e6ddbfed47f-metrics-cert\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.628013 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.627895    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71654660-e657-4ba8-9131-0e6ddbfed47f-cert\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.628013 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.627919    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppkf2\" (UniqueName: \"kubernetes.io/projected/71654660-e657-4ba8-9131-0e6ddbfed47f-kube-api-access-ppkf2\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.628630 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.628611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71654660-e657-4ba8-9131-0e6ddbfed47f-manager-config\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.630353 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.630332    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71654660-e657-4ba8-9131-0e6ddbfed47f-cert\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.630437 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.630387    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71654660-e657-4ba8-9131-0e6ddbfed47f-metrics-cert\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.636349 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.636328    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppkf2\" (UniqueName: \"kubernetes.io/projected/71654660-e657-4ba8-9131-0e6ddbfed47f-kube-api-access-ppkf2\") pod \"lws-controller-manager-f6d84646c-82tkk\" (UID: \"71654660-e657-4ba8-9131-0e6ddbfed47f\") " pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:14.779684 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:14.779612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:15.046922 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:15.046850    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"]
Sep 11 19:40:15.049642 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:40:15.049608    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71654660_e657_4ba8_9131_0e6ddbfed47f.slice/crio-34ba43c52c19411179dd7be4b7ea47d4e40c55d13172d2ece0504bc27252fa9b WatchSource:0}: Error finding container 34ba43c52c19411179dd7be4b7ea47d4e40c55d13172d2ece0504bc27252fa9b: Status 404 returned error can't find the container with id 34ba43c52c19411179dd7be4b7ea47d4e40c55d13172d2ece0504bc27252fa9b
Sep 11 19:40:15.151595 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:15.151560    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk" event={"ID":"71654660-e657-4ba8-9131-0e6ddbfed47f","Type":"ContainerStarted","Data":"34ba43c52c19411179dd7be4b7ea47d4e40c55d13172d2ece0504bc27252fa9b"}
Sep 11 19:40:17.162060 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:17.162029    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk" event={"ID":"71654660-e657-4ba8-9131-0e6ddbfed47f","Type":"ContainerStarted","Data":"354f699a7f4a777e135899b3201de5f05be29bd99f417236e721e03abaaa3ada"}
Sep 11 19:40:17.162439 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:17.162086    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:40:17.181092 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:17.181045    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk" podStartSLOduration=1.556358539 podStartE2EDuration="3.181029716s" totalImagesPullingTime="1.624671177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:17.179744454 +0000 UTC m=+653.855681367" watchObservedRunningTime="2026-09-11 19:40:17.181029716 +0000 UTC m=+653.856966626"
Sep 11 19:40:28.168985 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:40:28.168954    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f6d84646c-82tkk"
Sep 11 19:41:26.209004 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.208968    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"]
Sep 11 19:41:26.212090 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.212074    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.215166 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.215146    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 19:41:26.216325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.216307    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 19:41:26.216431 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.216339    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z9lkf\""
Sep 11 19:41:26.222599 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.222577    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"]
Sep 11 19:41:26.226186 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.226163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8fr\" (UniqueName: \"kubernetes.io/projected/9f091afc-420f-4c52-a07e-1f225ab0609c-kube-api-access-lt8fr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.226429 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.226210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.226429 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.226246    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.312804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.312770    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"]
Sep 11 19:41:26.315959 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.315939    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.324523 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.324498    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"]
Sep 11 19:41:26.327484 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.327462    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8fr\" (UniqueName: \"kubernetes.io/projected/9f091afc-420f-4c52-a07e-1f225ab0609c-kube-api-access-lt8fr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.327587 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.327498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.327587 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.327524    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.327703 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.327627    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4sv\" (UniqueName: \"kubernetes.io/projected/d646b013-ede8-4613-8777-12149eeccbf0-kube-api-access-kf4sv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.327703 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.327654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.327703 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.327673    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.327860 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.327832    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.327933 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.327917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.338021 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.337999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8fr\" (UniqueName: \"kubernetes.io/projected/9f091afc-420f-4c52-a07e-1f225ab0609c-kube-api-access-lt8fr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.409298 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.409273    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"]
Sep 11 19:41:26.418504 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.418477    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.424681 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.424652    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"]
Sep 11 19:41:26.428293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.428268    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.428402 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.428307    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4sv\" (UniqueName: \"kubernetes.io/projected/d646b013-ede8-4613-8777-12149eeccbf0-kube-api-access-kf4sv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.428402 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.428353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.428663 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.428643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.428748 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.428659    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.439467 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.439445    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4sv\" (UniqueName: \"kubernetes.io/projected/d646b013-ede8-4613-8777-12149eeccbf0-kube-api-access-kf4sv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.513484 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.513424    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"]
Sep 11 19:41:26.516634 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.516619    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.520444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.520422    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:26.527544 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.527521    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"]
Sep 11 19:41:26.530354 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.530331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.530471 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.530426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.530536 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.530470    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lc7g\" (UniqueName: \"kubernetes.io/projected/1eb0e568-befa-426b-83b7-97654029abce-kube-api-access-6lc7g\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.530536 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.530494    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.530536 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.530529    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.530684 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.530554    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jdz\" (UniqueName: \"kubernetes.io/projected/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-kube-api-access-p2jdz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.625290 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.625259    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:26.631468 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.631444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.631581 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.631476    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lc7g\" (UniqueName: \"kubernetes.io/projected/1eb0e568-befa-426b-83b7-97654029abce-kube-api-access-6lc7g\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.631581 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.631495    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.631704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.631609    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.631704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.631642    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jdz\" (UniqueName: \"kubernetes.io/projected/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-kube-api-access-p2jdz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.631704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.631669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.631907 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.631884    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.632053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.632032    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.632053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.632044    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.632180 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.632055    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.641075 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.641049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jdz\" (UniqueName: \"kubernetes.io/projected/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-kube-api-access-p2jdz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.641229 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.641211    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lc7g\" (UniqueName: \"kubernetes.io/projected/1eb0e568-befa-426b-83b7-97654029abce-kube-api-access-6lc7g\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.731003 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.730974    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:26.798748 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.798720    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"]
Sep 11 19:41:26.801662 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:41:26.801636    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f091afc_420f_4c52_a07e_1f225ab0609c.slice/crio-8053109ccc379a305524aa0c5c1887160c3353e00d35e47e87f74dfc3c9fcd50 WatchSource:0}: Error finding container 8053109ccc379a305524aa0c5c1887160c3353e00d35e47e87f74dfc3c9fcd50: Status 404 returned error can't find the container with id 8053109ccc379a305524aa0c5c1887160c3353e00d35e47e87f74dfc3c9fcd50
Sep 11 19:41:26.825478 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.825456    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:26.897814 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:26.897772    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"]
Sep 11 19:41:26.900836 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:41:26.900815    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd646b013_ede8_4613_8777_12149eeccbf0.slice/crio-48a5d764707d3dfd9595fbb4d83242f94443e6d1d65d1be7a528389fc6ec8481 WatchSource:0}: Error finding container 48a5d764707d3dfd9595fbb4d83242f94443e6d1d65d1be7a528389fc6ec8481: Status 404 returned error can't find the container with id 48a5d764707d3dfd9595fbb4d83242f94443e6d1d65d1be7a528389fc6ec8481
Sep 11 19:41:27.026991 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.026962    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"]
Sep 11 19:41:27.028131 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:41:27.028094    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb0e568_befa_426b_83b7_97654029abce.slice/crio-094ea3e20f40ac17bc4acb1cb82c34d75df21457d890618dc337cf9aae229982 WatchSource:0}: Error finding container 094ea3e20f40ac17bc4acb1cb82c34d75df21457d890618dc337cf9aae229982: Status 404 returned error can't find the container with id 094ea3e20f40ac17bc4acb1cb82c34d75df21457d890618dc337cf9aae229982
Sep 11 19:41:27.123011 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.122986    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"]
Sep 11 19:41:27.123458 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:41:27.123428    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2107ea_ac07_4b82_98df_9afc0f9c6e6b.slice/crio-0ae6c7d63378f5d19a96c26e823c3ac84015858be57afda3b02ff2517427d853 WatchSource:0}: Error finding container 0ae6c7d63378f5d19a96c26e823c3ac84015858be57afda3b02ff2517427d853: Status 404 returned error can't find the container with id 0ae6c7d63378f5d19a96c26e823c3ac84015858be57afda3b02ff2517427d853
Sep 11 19:41:27.454254 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.454217    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerID="8cacbfeae3377e7e25ae0d816b8d4b94da0626b597876780f26b6b96b58f693c" exitCode=0
Sep 11 19:41:27.454667 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.454301    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn" event={"ID":"9f091afc-420f-4c52-a07e-1f225ab0609c","Type":"ContainerDied","Data":"8cacbfeae3377e7e25ae0d816b8d4b94da0626b597876780f26b6b96b58f693c"}
Sep 11 19:41:27.454667 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.454352    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn" event={"ID":"9f091afc-420f-4c52-a07e-1f225ab0609c","Type":"ContainerStarted","Data":"8053109ccc379a305524aa0c5c1887160c3353e00d35e47e87f74dfc3c9fcd50"}
Sep 11 19:41:27.455439 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.455421    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:41:27.455939 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.455922    2571 generic.go:356] "Generic (PLEG): container finished" podID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerID="102ceebcf006653f75ddee98ed79db88741ee5d070eacbc668fe21764e803724" exitCode=0
Sep 11 19:41:27.456008 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.455992    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57" event={"ID":"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b","Type":"ContainerDied","Data":"102ceebcf006653f75ddee98ed79db88741ee5d070eacbc668fe21764e803724"}
Sep 11 19:41:27.456060 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.456033    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57" event={"ID":"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b","Type":"ContainerStarted","Data":"0ae6c7d63378f5d19a96c26e823c3ac84015858be57afda3b02ff2517427d853"}
Sep 11 19:41:27.457606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.457582    2571 generic.go:356] "Generic (PLEG): container finished" podID="1eb0e568-befa-426b-83b7-97654029abce" containerID="c6b750787a62cc91a920779d24929dcbe1f837af54c994566e77cff1799846ee" exitCode=0
Sep 11 19:41:27.457706 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.457667    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq" event={"ID":"1eb0e568-befa-426b-83b7-97654029abce","Type":"ContainerDied","Data":"c6b750787a62cc91a920779d24929dcbe1f837af54c994566e77cff1799846ee"}
Sep 11 19:41:27.457773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.457706    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq" event={"ID":"1eb0e568-befa-426b-83b7-97654029abce","Type":"ContainerStarted","Data":"094ea3e20f40ac17bc4acb1cb82c34d75df21457d890618dc337cf9aae229982"}
Sep 11 19:41:27.459969 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.459940    2571 generic.go:356] "Generic (PLEG): container finished" podID="d646b013-ede8-4613-8777-12149eeccbf0" containerID="aee2d40fcf1349e9ac8f7d49599540e9c27d54f4f3e4bc6f53218646c1303cae" exitCode=0
Sep 11 19:41:27.460073 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.459981    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l" event={"ID":"d646b013-ede8-4613-8777-12149eeccbf0","Type":"ContainerDied","Data":"aee2d40fcf1349e9ac8f7d49599540e9c27d54f4f3e4bc6f53218646c1303cae"}
Sep 11 19:41:27.460073 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:27.460016    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l" event={"ID":"d646b013-ede8-4613-8777-12149eeccbf0","Type":"ContainerStarted","Data":"48a5d764707d3dfd9595fbb4d83242f94443e6d1d65d1be7a528389fc6ec8481"}
Sep 11 19:41:28.465609 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:28.465573    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerID="8fca5798ee9b8604e36f54dafa1f164bd693e8f26deb1c2ae6badaaa06b2794f" exitCode=0
Sep 11 19:41:28.465942 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:28.465658    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn" event={"ID":"9f091afc-420f-4c52-a07e-1f225ab0609c","Type":"ContainerDied","Data":"8fca5798ee9b8604e36f54dafa1f164bd693e8f26deb1c2ae6badaaa06b2794f"}
Sep 11 19:41:28.467754 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:28.467732    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57" event={"ID":"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b","Type":"ContainerStarted","Data":"2dc53b10a73f43816917ed19ae8f72c41ebc60e7525f798ad05a663e135a9970"}
Sep 11 19:41:28.470186 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:28.470163    2571 generic.go:356] "Generic (PLEG): container finished" podID="1eb0e568-befa-426b-83b7-97654029abce" containerID="47cf13dce72297d31251ce2d1550fe9c129c24e5db471875df4dc960793c67d9" exitCode=0
Sep 11 19:41:28.470260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:28.470245    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq" event={"ID":"1eb0e568-befa-426b-83b7-97654029abce","Type":"ContainerDied","Data":"47cf13dce72297d31251ce2d1550fe9c129c24e5db471875df4dc960793c67d9"}
Sep 11 19:41:29.476844 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:29.476813    2571 generic.go:356] "Generic (PLEG): container finished" podID="d646b013-ede8-4613-8777-12149eeccbf0" containerID="0ffaab2ce0b35f4b4526754dabdac3b7dec15fe6c00d86a5d23be8f1a9745be1" exitCode=0
Sep 11 19:41:29.477219 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:29.476887    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l" event={"ID":"d646b013-ede8-4613-8777-12149eeccbf0","Type":"ContainerDied","Data":"0ffaab2ce0b35f4b4526754dabdac3b7dec15fe6c00d86a5d23be8f1a9745be1"}
Sep 11 19:41:29.479139 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:29.479114    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerID="173406a9213c2760008adcfc6286fd02ebf88a7e2a124ce7af2f5171e3997eba" exitCode=0
Sep 11 19:41:29.479220 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:29.479199    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn" event={"ID":"9f091afc-420f-4c52-a07e-1f225ab0609c","Type":"ContainerDied","Data":"173406a9213c2760008adcfc6286fd02ebf88a7e2a124ce7af2f5171e3997eba"}
Sep 11 19:41:29.480892 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:29.480868    2571 generic.go:356] "Generic (PLEG): container finished" podID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerID="2dc53b10a73f43816917ed19ae8f72c41ebc60e7525f798ad05a663e135a9970" exitCode=0
Sep 11 19:41:29.480988 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:29.480946    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57" event={"ID":"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b","Type":"ContainerDied","Data":"2dc53b10a73f43816917ed19ae8f72c41ebc60e7525f798ad05a663e135a9970"}
Sep 11 19:41:29.483294 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:29.483274    2571 generic.go:356] "Generic (PLEG): container finished" podID="1eb0e568-befa-426b-83b7-97654029abce" containerID="b79677d65b257fe0bb01617943cd9662f169560cb4204c1a7c95d7766f17d543" exitCode=0
Sep 11 19:41:29.483374 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:29.483324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq" event={"ID":"1eb0e568-befa-426b-83b7-97654029abce","Type":"ContainerDied","Data":"b79677d65b257fe0bb01617943cd9662f169560cb4204c1a7c95d7766f17d543"}
Sep 11 19:41:30.489383 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.489354    2571 generic.go:356] "Generic (PLEG): container finished" podID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerID="2aa4de4a03d75d495ac27032b617705d2dbe7439092f6c242ecd17d84b1eb12a" exitCode=0
Sep 11 19:41:30.489821 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.489440    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57" event={"ID":"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b","Type":"ContainerDied","Data":"2aa4de4a03d75d495ac27032b617705d2dbe7439092f6c242ecd17d84b1eb12a"}
Sep 11 19:41:30.491604 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.491581    2571 generic.go:356] "Generic (PLEG): container finished" podID="d646b013-ede8-4613-8777-12149eeccbf0" containerID="a9fc9b5ee2a178252bd52c75866d4363c4b3afd9e1c45a3850ac389a1d4ed94f" exitCode=0
Sep 11 19:41:30.491739 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.491648    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l" event={"ID":"d646b013-ede8-4613-8777-12149eeccbf0","Type":"ContainerDied","Data":"a9fc9b5ee2a178252bd52c75866d4363c4b3afd9e1c45a3850ac389a1d4ed94f"}
Sep 11 19:41:30.618187 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.618164    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:30.645425 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.645399    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:30.658193 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.658172    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8fr\" (UniqueName: \"kubernetes.io/projected/9f091afc-420f-4c52-a07e-1f225ab0609c-kube-api-access-lt8fr\") pod \"9f091afc-420f-4c52-a07e-1f225ab0609c\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") "
Sep 11 19:41:30.658294 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.658205    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-bundle\") pod \"9f091afc-420f-4c52-a07e-1f225ab0609c\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") "
Sep 11 19:41:30.658294 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.658243    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-util\") pod \"9f091afc-420f-4c52-a07e-1f225ab0609c\" (UID: \"9f091afc-420f-4c52-a07e-1f225ab0609c\") "
Sep 11 19:41:30.658294 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.658272    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lc7g\" (UniqueName: \"kubernetes.io/projected/1eb0e568-befa-426b-83b7-97654029abce-kube-api-access-6lc7g\") pod \"1eb0e568-befa-426b-83b7-97654029abce\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") "
Sep 11 19:41:30.658294 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.658289    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-bundle\") pod \"1eb0e568-befa-426b-83b7-97654029abce\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") "
Sep 11 19:41:30.658488 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.658303    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-util\") pod \"1eb0e568-befa-426b-83b7-97654029abce\" (UID: \"1eb0e568-befa-426b-83b7-97654029abce\") "
Sep 11 19:41:30.658716 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.658688    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-bundle" (OuterVolumeSpecName: "bundle") pod "9f091afc-420f-4c52-a07e-1f225ab0609c" (UID: "9f091afc-420f-4c52-a07e-1f225ab0609c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:41:30.659487 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.659461    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-bundle" (OuterVolumeSpecName: "bundle") pod "1eb0e568-befa-426b-83b7-97654029abce" (UID: "1eb0e568-befa-426b-83b7-97654029abce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:41:30.660393 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.660364    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f091afc-420f-4c52-a07e-1f225ab0609c-kube-api-access-lt8fr" (OuterVolumeSpecName: "kube-api-access-lt8fr") pod "9f091afc-420f-4c52-a07e-1f225ab0609c" (UID: "9f091afc-420f-4c52-a07e-1f225ab0609c"). InnerVolumeSpecName "kube-api-access-lt8fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:41:30.660765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.660742    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb0e568-befa-426b-83b7-97654029abce-kube-api-access-6lc7g" (OuterVolumeSpecName: "kube-api-access-6lc7g") pod "1eb0e568-befa-426b-83b7-97654029abce" (UID: "1eb0e568-befa-426b-83b7-97654029abce"). InnerVolumeSpecName "kube-api-access-6lc7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:41:30.663639 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.663614    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-util" (OuterVolumeSpecName: "util") pod "1eb0e568-befa-426b-83b7-97654029abce" (UID: "1eb0e568-befa-426b-83b7-97654029abce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:41:30.664604 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.664570    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-util" (OuterVolumeSpecName: "util") pod "9f091afc-420f-4c52-a07e-1f225ab0609c" (UID: "9f091afc-420f-4c52-a07e-1f225ab0609c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:41:30.759125 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.759043    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8fr\" (UniqueName: \"kubernetes.io/projected/9f091afc-420f-4c52-a07e-1f225ab0609c-kube-api-access-lt8fr\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:30.759125 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.759066    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-bundle\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:30.759125 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.759085    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f091afc-420f-4c52-a07e-1f225ab0609c-util\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:30.759125 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.759096    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lc7g\" (UniqueName: \"kubernetes.io/projected/1eb0e568-befa-426b-83b7-97654029abce-kube-api-access-6lc7g\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:30.759125 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.759124    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-bundle\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:30.759350 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:30.759131    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb0e568-befa-426b-83b7-97654029abce-util\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:31.498335 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.498302    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq"
Sep 11 19:41:31.498335 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.498325    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2g2xq" event={"ID":"1eb0e568-befa-426b-83b7-97654029abce","Type":"ContainerDied","Data":"094ea3e20f40ac17bc4acb1cb82c34d75df21457d890618dc337cf9aae229982"}
Sep 11 19:41:31.498839 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.498369    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094ea3e20f40ac17bc4acb1cb82c34d75df21457d890618dc337cf9aae229982"
Sep 11 19:41:31.500264 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.500234    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn" event={"ID":"9f091afc-420f-4c52-a07e-1f225ab0609c","Type":"ContainerDied","Data":"8053109ccc379a305524aa0c5c1887160c3353e00d35e47e87f74dfc3c9fcd50"}
Sep 11 19:41:31.500363 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.500272    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdkhn"
Sep 11 19:41:31.500363 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.500280    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8053109ccc379a305524aa0c5c1887160c3353e00d35e47e87f74dfc3c9fcd50"
Sep 11 19:41:31.624984 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.624952    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:31.661682 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.661659    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:41:31.664648 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.664632    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-bundle\") pod \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") "
Sep 11 19:41:31.664713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.664661    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2jdz\" (UniqueName: \"kubernetes.io/projected/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-kube-api-access-p2jdz\") pod \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") "
Sep 11 19:41:31.664713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.664678    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-util\") pod \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\" (UID: \"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b\") "
Sep 11 19:41:31.664792 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.664725    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-bundle\") pod \"d646b013-ede8-4613-8777-12149eeccbf0\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") "
Sep 11 19:41:31.664792 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.664752    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-util\") pod \"d646b013-ede8-4613-8777-12149eeccbf0\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") "
Sep 11 19:41:31.664792 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.664779    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf4sv\" (UniqueName: \"kubernetes.io/projected/d646b013-ede8-4613-8777-12149eeccbf0-kube-api-access-kf4sv\") pod \"d646b013-ede8-4613-8777-12149eeccbf0\" (UID: \"d646b013-ede8-4613-8777-12149eeccbf0\") "
Sep 11 19:41:31.665207 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.665180    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-bundle" (OuterVolumeSpecName: "bundle") pod "bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" (UID: "bc2107ea-ac07-4b82-98df-9afc0f9c6e6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:41:31.665299 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.665285    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-bundle" (OuterVolumeSpecName: "bundle") pod "d646b013-ede8-4613-8777-12149eeccbf0" (UID: "d646b013-ede8-4613-8777-12149eeccbf0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:41:31.666703 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.666680    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-kube-api-access-p2jdz" (OuterVolumeSpecName: "kube-api-access-p2jdz") pod "bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" (UID: "bc2107ea-ac07-4b82-98df-9afc0f9c6e6b"). InnerVolumeSpecName "kube-api-access-p2jdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:41:31.666991 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.666969    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d646b013-ede8-4613-8777-12149eeccbf0-kube-api-access-kf4sv" (OuterVolumeSpecName: "kube-api-access-kf4sv") pod "d646b013-ede8-4613-8777-12149eeccbf0" (UID: "d646b013-ede8-4613-8777-12149eeccbf0"). InnerVolumeSpecName "kube-api-access-kf4sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:41:31.671424 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.671395    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-util" (OuterVolumeSpecName: "util") pod "d646b013-ede8-4613-8777-12149eeccbf0" (UID: "d646b013-ede8-4613-8777-12149eeccbf0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:41:31.672136 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.672095    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-util" (OuterVolumeSpecName: "util") pod "bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" (UID: "bc2107ea-ac07-4b82-98df-9afc0f9c6e6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:41:31.766175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.766095    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-bundle\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:31.766175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.766135    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d646b013-ede8-4613-8777-12149eeccbf0-util\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:31.766175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.766144    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf4sv\" (UniqueName: \"kubernetes.io/projected/d646b013-ede8-4613-8777-12149eeccbf0-kube-api-access-kf4sv\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:31.766175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.766154    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-bundle\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:31.766175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.766163    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jdz\" (UniqueName: \"kubernetes.io/projected/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-kube-api-access-p2jdz\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:31.766175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:31.766170    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc2107ea-ac07-4b82-98df-9afc0f9c6e6b-util\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:41:32.506875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:32.506847    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57"
Sep 11 19:41:32.507261 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:32.506886    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c646f57" event={"ID":"bc2107ea-ac07-4b82-98df-9afc0f9c6e6b","Type":"ContainerDied","Data":"0ae6c7d63378f5d19a96c26e823c3ac84015858be57afda3b02ff2517427d853"}
Sep 11 19:41:32.507261 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:32.506930    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ae6c7d63378f5d19a96c26e823c3ac84015858be57afda3b02ff2517427d853"
Sep 11 19:41:32.509025 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:32.509003    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l" event={"ID":"d646b013-ede8-4613-8777-12149eeccbf0","Type":"ContainerDied","Data":"48a5d764707d3dfd9595fbb4d83242f94443e6d1d65d1be7a528389fc6ec8481"}
Sep 11 19:41:32.509154 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:32.509042    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a5d764707d3dfd9595fbb4d83242f94443e6d1d65d1be7a528389fc6ec8481"
Sep 11 19:41:32.509154 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:41:32.509051    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl7n7l"
Sep 11 19:42:35.705665 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705635    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705870    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705880    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705889    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d646b013-ede8-4613-8777-12149eeccbf0" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705894    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d646b013-ede8-4613-8777-12149eeccbf0" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705901    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerName="pull"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705906    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerName="pull"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705913    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705918    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705928    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerName="util"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705933    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerName="util"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705940    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d646b013-ede8-4613-8777-12149eeccbf0" containerName="util"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705945    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d646b013-ede8-4613-8777-12149eeccbf0" containerName="util"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705955    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d646b013-ede8-4613-8777-12149eeccbf0" containerName="pull"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705959    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d646b013-ede8-4613-8777-12149eeccbf0" containerName="pull"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705965    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerName="util"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705970    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerName="util"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705975    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerName="pull"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705980    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerName="pull"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705985    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb0e568-befa-426b-83b7-97654029abce" containerName="pull"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705990    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb0e568-befa-426b-83b7-97654029abce" containerName="pull"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.705998    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb0e568-befa-426b-83b7-97654029abce" containerName="util"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.706003    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb0e568-befa-426b-83b7-97654029abce" containerName="util"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.706007    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb0e568-befa-426b-83b7-97654029abce" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.706013    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb0e568-befa-426b-83b7-97654029abce" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.706053    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2107ea-ac07-4b82-98df-9afc0f9c6e6b" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.706060    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb0e568-befa-426b-83b7-97654029abce" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.706068    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d646b013-ede8-4613-8777-12149eeccbf0" containerName="extract"
Sep 11 19:42:35.706131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.706073    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f091afc-420f-4c52-a07e-1f225ab0609c" containerName="extract"
Sep 11 19:42:35.708801 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.708784    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:35.711785 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.711761    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 19:42:35.711890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.711840    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9z8fl\""
Sep 11 19:42:35.712827 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.712809    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 19:42:35.712941 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.712921    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 19:42:35.719519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.719498    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:42:35.763978 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.763949    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:42:35.804410 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.804385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ct6\" (UniqueName: \"kubernetes.io/projected/56d7065b-4d2b-4705-98eb-30a3ad5e0ee6-kube-api-access-t7ct6\") pod \"limitador-limitador-84994f7ffc-tb4n9\" (UID: \"56d7065b-4d2b-4705-98eb-30a3ad5e0ee6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:35.804556 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.804439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56d7065b-4d2b-4705-98eb-30a3ad5e0ee6-config-file\") pod \"limitador-limitador-84994f7ffc-tb4n9\" (UID: \"56d7065b-4d2b-4705-98eb-30a3ad5e0ee6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:35.905233 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.905208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56d7065b-4d2b-4705-98eb-30a3ad5e0ee6-config-file\") pod \"limitador-limitador-84994f7ffc-tb4n9\" (UID: \"56d7065b-4d2b-4705-98eb-30a3ad5e0ee6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:35.905391 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.905255    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ct6\" (UniqueName: \"kubernetes.io/projected/56d7065b-4d2b-4705-98eb-30a3ad5e0ee6-kube-api-access-t7ct6\") pod \"limitador-limitador-84994f7ffc-tb4n9\" (UID: \"56d7065b-4d2b-4705-98eb-30a3ad5e0ee6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:35.905831 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.905808    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56d7065b-4d2b-4705-98eb-30a3ad5e0ee6-config-file\") pod \"limitador-limitador-84994f7ffc-tb4n9\" (UID: \"56d7065b-4d2b-4705-98eb-30a3ad5e0ee6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:35.913881 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:35.913853    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ct6\" (UniqueName: \"kubernetes.io/projected/56d7065b-4d2b-4705-98eb-30a3ad5e0ee6-kube-api-access-t7ct6\") pod \"limitador-limitador-84994f7ffc-tb4n9\" (UID: \"56d7065b-4d2b-4705-98eb-30a3ad5e0ee6\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:36.019520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:36.019456    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:36.293724 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:36.293702    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:42:36.295452 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:42:36.295422    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d7065b_4d2b_4705_98eb_30a3ad5e0ee6.slice/crio-a85d9505a2d9c10bea9f3ddae5f3f7216eb09588305e8695cbc52abd2f27a9d7 WatchSource:0}: Error finding container a85d9505a2d9c10bea9f3ddae5f3f7216eb09588305e8695cbc52abd2f27a9d7: Status 404 returned error can't find the container with id a85d9505a2d9c10bea9f3ddae5f3f7216eb09588305e8695cbc52abd2f27a9d7
Sep 11 19:42:36.791001 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:36.790964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9" event={"ID":"56d7065b-4d2b-4705-98eb-30a3ad5e0ee6","Type":"ContainerStarted","Data":"a85d9505a2d9c10bea9f3ddae5f3f7216eb09588305e8695cbc52abd2f27a9d7"}
Sep 11 19:42:38.802590 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:38.802508    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9" event={"ID":"56d7065b-4d2b-4705-98eb-30a3ad5e0ee6","Type":"ContainerStarted","Data":"2e7166bec6530b659f8f620511648746e11b340d3ebebea4d580bccee17f0dfd"}
Sep 11 19:42:38.802953 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:38.802655    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:42:38.821444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:38.821395    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9" podStartSLOduration=1.5751086760000002 podStartE2EDuration="3.821381762s" totalImagesPullingTime="2.246273086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:42:38.819893476 +0000 UTC m=+795.495830400" watchObservedRunningTime="2026-09-11 19:42:38.821381762 +0000 UTC m=+795.497318669"
Sep 11 19:42:49.808277 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:42:49.808201    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"
Sep 11 19:43:43.590912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:43:43.590878    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:43:59.657461 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:43:59.657427    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:44:13.945862 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.945831    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"]
Sep 11 19:44:13.954404 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.954381    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:13.957289 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.957267    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 19:44:13.958467 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.958449    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 19:44:13.958467 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.958459    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 19:44:13.958650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.958566    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vwxjz\""
Sep 11 19:44:13.970606 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.970585    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"]
Sep 11 19:44:13.980967 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.980939    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:13.981056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.980980    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:13.981056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.981017    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbtp\" (UniqueName: \"kubernetes.io/projected/6db52705-77cf-49c1-bf71-98f8cc5dfe30-kube-api-access-rmbtp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:13.981167 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.981072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:13.981167 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.981157    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:13.981239 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:13.981182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db52705-77cf-49c1-bf71-98f8cc5dfe30-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082163 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082142    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082181    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082201    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db52705-77cf-49c1-bf71-98f8cc5dfe30-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbtp\" (UniqueName: \"kubernetes.io/projected/6db52705-77cf-49c1-bf71-98f8cc5dfe30-kube-api-access-rmbtp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082691 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082652    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082691 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082677    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.082829 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.082707    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.084360 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.084335    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db52705-77cf-49c1-bf71-98f8cc5dfe30-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.084592 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.084573    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db52705-77cf-49c1-bf71-98f8cc5dfe30-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.090914 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.090893    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbtp\" (UniqueName: \"kubernetes.io/projected/6db52705-77cf-49c1-bf71-98f8cc5dfe30-kube-api-access-rmbtp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s8hpw\" (UID: \"6db52705-77cf-49c1-bf71-98f8cc5dfe30\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.287083 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.287018    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:14.558807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.558704    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"]
Sep 11 19:44:14.561394 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:44:14.561357    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db52705_77cf_49c1_bf71_98f8cc5dfe30.slice/crio-2be728f3f8ee10c911aa0ae4bf19f7a0260b2a24a3196cd2eba79b37790d8870 WatchSource:0}: Error finding container 2be728f3f8ee10c911aa0ae4bf19f7a0260b2a24a3196cd2eba79b37790d8870: Status 404 returned error can't find the container with id 2be728f3f8ee10c911aa0ae4bf19f7a0260b2a24a3196cd2eba79b37790d8870
Sep 11 19:44:14.659665 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:14.659640    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:44:15.244964 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:15.244775    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw" event={"ID":"6db52705-77cf-49c1-bf71-98f8cc5dfe30","Type":"ContainerStarted","Data":"2be728f3f8ee10c911aa0ae4bf19f7a0260b2a24a3196cd2eba79b37790d8870"}
Sep 11 19:44:16.250324 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:16.250239    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw" event={"ID":"6db52705-77cf-49c1-bf71-98f8cc5dfe30","Type":"ContainerStarted","Data":"1c493f561d961f0efde4b1a89b31abd2bae0065a67082472a36bd882989a396d"}
Sep 11 19:44:16.273165 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:16.273112    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw" podStartSLOduration=2.276443221 podStartE2EDuration="3.273082942s" totalImagesPullingTime="996.639721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:16.271586609 +0000 UTC m=+892.947523514" watchObservedRunningTime="2026-09-11 19:44:16.273082942 +0000 UTC m=+892.949019850"
Sep 11 19:44:23.452490 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:23.452450    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:44:24.288002 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:24.287972    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:24.288223 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:24.288055    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:24.300595 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:24.300574    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:25.301209 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:25.301177    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s8hpw"
Sep 11 19:44:28.956240 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:28.956208    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:44:35.456097 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:35.456054    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:44:51.970931 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:51.970899    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"]
Sep 11 19:44:51.975016 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:51.974997    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:51.978432 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:51.978405    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 19:44:51.990347 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:51.990326    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"]
Sep 11 19:44:52.052476 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.052455    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da82de53-777b-471a-b96b-adc1519fc6f8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.052579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.052485    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gbw\" (UniqueName: \"kubernetes.io/projected/da82de53-777b-471a-b96b-adc1519fc6f8-kube-api-access-t2gbw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.052579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.052508    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.052579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.052528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.052686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.052599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.052686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.052637    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.152902 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.152874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.153000 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.152910    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da82de53-777b-471a-b96b-adc1519fc6f8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.153000 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.152929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gbw\" (UniqueName: \"kubernetes.io/projected/da82de53-777b-471a-b96b-adc1519fc6f8-kube-api-access-t2gbw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.153000 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.152950    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.153149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.153078    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.153200 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.153169    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.153315 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.153296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.153399 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.153335    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.153515 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.153496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.155177 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.155148    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da82de53-777b-471a-b96b-adc1519fc6f8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.155405 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.155388    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da82de53-777b-471a-b96b-adc1519fc6f8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.163282 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.163264    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gbw\" (UniqueName: \"kubernetes.io/projected/da82de53-777b-471a-b96b-adc1519fc6f8-kube-api-access-t2gbw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmc5h\" (UID: \"da82de53-777b-471a-b96b-adc1519fc6f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.284078 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.284015    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:44:52.555924 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:52.555895    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"]
Sep 11 19:44:52.557767 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:44:52.557742    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda82de53_777b_471a_b96b_adc1519fc6f8.slice/crio-b142e4da6562df13f1165efe1507f1e95fdb4c71d3db5cc75537144cbcb7ae87 WatchSource:0}: Error finding container b142e4da6562df13f1165efe1507f1e95fdb4c71d3db5cc75537144cbcb7ae87: Status 404 returned error can't find the container with id b142e4da6562df13f1165efe1507f1e95fdb4c71d3db5cc75537144cbcb7ae87
Sep 11 19:44:53.416163 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:53.416125    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h" event={"ID":"da82de53-777b-471a-b96b-adc1519fc6f8","Type":"ContainerStarted","Data":"d46372aefd75694519fff053aee33a729d51820206f2dd37c370cdb809681542"}
Sep 11 19:44:53.416529 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:53.416183    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h" event={"ID":"da82de53-777b-471a-b96b-adc1519fc6f8","Type":"ContainerStarted","Data":"b142e4da6562df13f1165efe1507f1e95fdb4c71d3db5cc75537144cbcb7ae87"}
Sep 11 19:44:53.439993 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:53.439947    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h" podStartSLOduration=2.273136643 podStartE2EDuration="2.439930861s" totalImagesPullingTime="166.794218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:53.437510289 +0000 UTC m=+930.113447197" watchObservedRunningTime="2026-09-11 19:44:53.439930861 +0000 UTC m=+930.115867768"
Sep 11 19:44:53.858830 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:53.858755    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:44:58.550846 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.550813    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"]
Sep 11 19:44:58.554317 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.554301    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.557468 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.557443    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 19:44:58.569747 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.569726    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"]
Sep 11 19:44:58.702375 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.702340    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.702537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.702389    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.702537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.702480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4e3640-c581-4541-a863-e5c00c26c30c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.702537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.702510    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8xx\" (UniqueName: \"kubernetes.io/projected/bc4e3640-c581-4541-a863-e5c00c26c30c-kube-api-access-hd8xx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.702537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.702534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.702673 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.702568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.803635 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.803567    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.803635 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.803610    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.803635 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.803628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.803865 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.803653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.803865 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.803689    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4e3640-c581-4541-a863-e5c00c26c30c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.803865 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.803705    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8xx\" (UniqueName: \"kubernetes.io/projected/bc4e3640-c581-4541-a863-e5c00c26c30c-kube-api-access-hd8xx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.804019 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.803981    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.804075 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.804026    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.804075 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.804056    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.805984 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.805962    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4e3640-c581-4541-a863-e5c00c26c30c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.806219 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.806199    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4e3640-c581-4541-a863-e5c00c26c30c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.823391 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.823363    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8xx\" (UniqueName: \"kubernetes.io/projected/bc4e3640-c581-4541-a863-e5c00c26c30c-kube-api-access-hd8xx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7\" (UID: \"bc4e3640-c581-4541-a863-e5c00c26c30c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:58.863367 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:58.863345    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:44:59.135750 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:59.135722    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"]
Sep 11 19:44:59.137375 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:44:59.137350    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc4e3640_c581_4541_a863_e5c00c26c30c.slice/crio-850cdc7f9465c5ef6a120acf210206bc1ba97f7a1831d89d9f064512459bae8a WatchSource:0}: Error finding container 850cdc7f9465c5ef6a120acf210206bc1ba97f7a1831d89d9f064512459bae8a: Status 404 returned error can't find the container with id 850cdc7f9465c5ef6a120acf210206bc1ba97f7a1831d89d9f064512459bae8a
Sep 11 19:44:59.452573 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:59.452485    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7" event={"ID":"bc4e3640-c581-4541-a863-e5c00c26c30c","Type":"ContainerStarted","Data":"da523d6e4c5f690977ef1624b35e84d87257e65a5f728bd0d285e06e006d08fa"}
Sep 11 19:44:59.452573 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:59.452535    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7" event={"ID":"bc4e3640-c581-4541-a863-e5c00c26c30c","Type":"ContainerStarted","Data":"850cdc7f9465c5ef6a120acf210206bc1ba97f7a1831d89d9f064512459bae8a"}
Sep 11 19:44:59.475235 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:44:59.475190    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7" podStartSLOduration=1.311144802 podStartE2EDuration="1.475175829s" totalImagesPullingTime="164.031027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:59.472401201 +0000 UTC m=+936.148338109" watchObservedRunningTime="2026-09-11 19:44:59.475175829 +0000 UTC m=+936.151112736"
Sep 11 19:45:00.149876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:00.149841    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb"]
Sep 11 19:45:00.153056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:00.153039    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb"
Sep 11 19:45:00.157081 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:00.157064    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 19:45:00.162319 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:00.162299    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb"
Sep 11 19:45:00.172134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:00.172112    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb"]
Sep 11 19:45:00.434057 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:00.434028    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb"]
Sep 11 19:45:00.435823 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:45:00.435799    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007ba46e_aa9e_4dfc_9ca2_6c77bf626bb9.slice/crio-9fe896b141eb4a80d67055ee145ce070fabb2e0f603a0b6febc1b7615abd2141 WatchSource:0}: Error finding container 9fe896b141eb4a80d67055ee145ce070fabb2e0f603a0b6febc1b7615abd2141: Status 404 returned error can't find the container with id 9fe896b141eb4a80d67055ee145ce070fabb2e0f603a0b6febc1b7615abd2141
Sep 11 19:45:00.458263 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:00.458239    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb" event={"ID":"007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9","Type":"ContainerStarted","Data":"9fe896b141eb4a80d67055ee145ce070fabb2e0f603a0b6febc1b7615abd2141"}
Sep 11 19:45:01.767726 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:01.767693    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:45:02.285244 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:02.285208    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:45:02.285420 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:02.285282    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:45:02.299487 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:02.299462    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:45:02.480713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:02.480681    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmc5h"
Sep 11 19:45:03.474707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:03.474670    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb" event={"ID":"007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9","Type":"ContainerStarted","Data":"8253c69f6ece123878030f8c99b3b4ed947bac049233698bf6d465c9c7b195a3"}
Sep 11 19:45:03.501963 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:03.501914    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb" podStartSLOduration=1.129722243 podStartE2EDuration="3.501900492s" totalImagesPullingTime="2.372178249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:03.499318152 +0000 UTC m=+940.175255055" watchObservedRunningTime="2026-09-11 19:45:03.501900492 +0000 UTC m=+940.177837402"
Sep 11 19:45:04.481084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:04.481050    2571 generic.go:356] "Generic (PLEG): container finished" podID="007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9" containerID="8253c69f6ece123878030f8c99b3b4ed947bac049233698bf6d465c9c7b195a3" exitCode=0
Sep 11 19:45:04.481447 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:04.481137    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb" event={"ID":"007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9","Type":"ContainerDied","Data":"8253c69f6ece123878030f8c99b3b4ed947bac049233698bf6d465c9c7b195a3"}
Sep 11 19:45:05.603774 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:05.603752    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb"
Sep 11 19:45:06.491360 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:06.491332    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb"
Sep 11 19:45:06.491515 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:06.491359    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-9wrsb" event={"ID":"007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9","Type":"ContainerDied","Data":"9fe896b141eb4a80d67055ee145ce070fabb2e0f603a0b6febc1b7615abd2141"}
Sep 11 19:45:06.491515 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:06.491401    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fe896b141eb4a80d67055ee145ce070fabb2e0f603a0b6febc1b7615abd2141"
Sep 11 19:45:08.863683 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:08.863651    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:45:08.864125 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:08.863703    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:45:08.875942 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:08.875916    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:45:09.516367 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:45:09.516336    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7"
Sep 11 19:46:47.734449 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.734409    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7509-85bba4"]
Sep 11 19:46:47.734943 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.734676    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9" containerName="cleanup"
Sep 11 19:46:47.734943 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.734687    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9" containerName="cleanup"
Sep 11 19:46:47.734943 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.734746    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9" containerName="cleanup"
Sep 11 19:46:47.737548 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.737532    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-85bba4"
Sep 11 19:46:47.740446 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.740420    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-76rlb\""
Sep 11 19:46:47.745678 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.745654    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-85bba4"]
Sep 11 19:46:47.863281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.863245    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2wj\" (UniqueName: \"kubernetes.io/projected/c648c2ad-0204-47af-a679-56ecaf110eae-kube-api-access-mj2wj\") pod \"test-curl-7509-85bba4\" (UID: \"c648c2ad-0204-47af-a679-56ecaf110eae\") " pod="opendatahub/test-curl-7509-85bba4"
Sep 11 19:46:47.964578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.964549    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2wj\" (UniqueName: \"kubernetes.io/projected/c648c2ad-0204-47af-a679-56ecaf110eae-kube-api-access-mj2wj\") pod \"test-curl-7509-85bba4\" (UID: \"c648c2ad-0204-47af-a679-56ecaf110eae\") " pod="opendatahub/test-curl-7509-85bba4"
Sep 11 19:46:47.973801 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:47.973771    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2wj\" (UniqueName: \"kubernetes.io/projected/c648c2ad-0204-47af-a679-56ecaf110eae-kube-api-access-mj2wj\") pod \"test-curl-7509-85bba4\" (UID: \"c648c2ad-0204-47af-a679-56ecaf110eae\") " pod="opendatahub/test-curl-7509-85bba4"
Sep 11 19:46:48.046825 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:48.046761    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-85bba4"
Sep 11 19:46:48.318982 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:48.318953    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-85bba4"]
Sep 11 19:46:48.321387 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:46:48.321356    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc648c2ad_0204_47af_a679_56ecaf110eae.slice/crio-38e8474a4fa19f8154ca6417c6f64bc97d4d797d30f3444b7c4e76ed68137f79 WatchSource:0}: Error finding container 38e8474a4fa19f8154ca6417c6f64bc97d4d797d30f3444b7c4e76ed68137f79: Status 404 returned error can't find the container with id 38e8474a4fa19f8154ca6417c6f64bc97d4d797d30f3444b7c4e76ed68137f79
Sep 11 19:46:48.323285 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:48.323263    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:46:48.959949 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:48.959919    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-85bba4" event={"ID":"c648c2ad-0204-47af-a679-56ecaf110eae","Type":"ContainerStarted","Data":"38e8474a4fa19f8154ca6417c6f64bc97d4d797d30f3444b7c4e76ed68137f79"}
Sep 11 19:46:50.283709 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:50.283617    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:46:50.973468 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:50.973434    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-85bba4" event={"ID":"c648c2ad-0204-47af-a679-56ecaf110eae","Type":"ContainerStarted","Data":"b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13"}
Sep 11 19:46:50.997454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:50.997407    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7509-85bba4" podStartSLOduration=1.848515512 podStartE2EDuration="3.997392045s" totalImagesPullingTime="2.148876533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:50.997361557 +0000 UTC m=+1047.673298466" watchObservedRunningTime="2026-09-11 19:46:50.997392045 +0000 UTC m=+1047.673328953"
Sep 11 19:46:51.634905 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.634870    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7509-85bba4"]
Sep 11 19:46:51.645269 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.645244    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7509-85bba4"]
Sep 11 19:46:51.766891 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.766824    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7509-45945f"]
Sep 11 19:46:51.767142 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.767128    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c648c2ad-0204-47af-a679-56ecaf110eae" containerName="test-curl-7509-85bba4"
Sep 11 19:46:51.767191 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.767144    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c648c2ad-0204-47af-a679-56ecaf110eae" containerName="test-curl-7509-85bba4"
Sep 11 19:46:51.767224 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.767203    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c648c2ad-0204-47af-a679-56ecaf110eae" containerName="test-curl-7509-85bba4"
Sep 11 19:46:51.769628 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.769613    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-45945f"
Sep 11 19:46:51.778910 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.778886    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-45945f"]
Sep 11 19:46:51.896301 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.896276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwmq\" (UniqueName: \"kubernetes.io/projected/c336f852-881b-4cf2-9f90-4f499551b5cf-kube-api-access-bfwmq\") pod \"test-curl-7509-45945f\" (UID: \"c336f852-881b-4cf2-9f90-4f499551b5cf\") " pod="opendatahub/test-curl-7509-45945f"
Sep 11 19:46:51.979165 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:46:51.979136    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7509-85bba4" container="test-curl-7509-85bba4"
Sep 11 19:46:51.997495 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:51.997466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwmq\" (UniqueName: \"kubernetes.io/projected/c336f852-881b-4cf2-9f90-4f499551b5cf-kube-api-access-bfwmq\") pod \"test-curl-7509-45945f\" (UID: \"c336f852-881b-4cf2-9f90-4f499551b5cf\") " pod="opendatahub/test-curl-7509-45945f"
Sep 11 19:46:52.008874 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:52.008853    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwmq\" (UniqueName: \"kubernetes.io/projected/c336f852-881b-4cf2-9f90-4f499551b5cf-kube-api-access-bfwmq\") pod \"test-curl-7509-45945f\" (UID: \"c336f852-881b-4cf2-9f90-4f499551b5cf\") " pod="opendatahub/test-curl-7509-45945f"
Sep 11 19:46:52.078594 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:52.078538    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-45945f"
Sep 11 19:46:52.367013 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:52.364879    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7509-45945f"]
Sep 11 19:46:52.368866 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:46:52.368836    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc336f852_881b_4cf2_9f90_4f499551b5cf.slice/crio-201a229462f79c229b6c937015f13bac2f34ef8eca8a0fd011d7dff9ee8eff6b WatchSource:0}: Error finding container 201a229462f79c229b6c937015f13bac2f34ef8eca8a0fd011d7dff9ee8eff6b: Status 404 returned error can't find the container with id 201a229462f79c229b6c937015f13bac2f34ef8eca8a0fd011d7dff9ee8eff6b
Sep 11 19:46:52.984728 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:52.984689    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-45945f" event={"ID":"c336f852-881b-4cf2-9f90-4f499551b5cf","Type":"ContainerStarted","Data":"40ce13e9c825eec133604d7946bb046ebf3082e01ca84126b03f8b7e47f8cb1b"}
Sep 11 19:46:52.985185 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:52.984736    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7509-45945f" event={"ID":"c336f852-881b-4cf2-9f90-4f499551b5cf","Type":"ContainerStarted","Data":"201a229462f79c229b6c937015f13bac2f34ef8eca8a0fd011d7dff9ee8eff6b"}
Sep 11 19:46:52.985185 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:52.984810    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7509-85bba4" podUID="c648c2ad-0204-47af-a679-56ecaf110eae" containerName="test-curl-7509-85bba4" containerID="cri-o://b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13" gracePeriod=2
Sep 11 19:46:53.030262 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:53.030215    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7509-45945f" podStartSLOduration=2.030197492 podStartE2EDuration="2.030197492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:53.027682813 +0000 UTC m=+1049.703619721" watchObservedRunningTime="2026-09-11 19:46:53.030197492 +0000 UTC m=+1049.706134402"
Sep 11 19:46:53.705374 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:53.705298    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7509-45945f"]
Sep 11 19:46:53.710332 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:53.710305    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7509-45945f"]
Sep 11 19:46:53.989157 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:46:53.989069    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7509-45945f" container="test-curl-7509-45945f"
Sep 11 19:46:54.660760 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.660719    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"]
Sep 11 19:46:54.661066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.661049    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c336f852-881b-4cf2-9f90-4f499551b5cf" containerName="test-curl-7509-45945f"
Sep 11 19:46:54.661171 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.661068    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c336f852-881b-4cf2-9f90-4f499551b5cf" containerName="test-curl-7509-45945f"
Sep 11 19:46:54.661230 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.661172    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c336f852-881b-4cf2-9f90-4f499551b5cf" containerName="test-curl-7509-45945f"
Sep 11 19:46:54.663988 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.663967    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.667565 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.667544    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 19:46:54.667641 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.667625    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7df2ac-openshift-default-dockercfg-kvdzx\""
Sep 11 19:46:54.689308 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.689287    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"]
Sep 11 19:46:54.718685 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.718658    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975837cd-aff1-404b-a486-d8856ce8ef11-istio-podinfo\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.718786 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.718691    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975837cd-aff1-404b-a486-d8856ce8ef11-istiod-ca-cert\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.718786 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.718727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-certs\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.718870 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.718844    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-socket\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.718914 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.718878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-envoy\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.718989 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.718969    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-istio-token\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.719048 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.719013    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-credential-socket\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.719048 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.719039    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-data\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.719175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.719070    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwx4q\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-kube-api-access-rwx4q\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820445 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975837cd-aff1-404b-a486-d8856ce8ef11-istio-podinfo\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820445 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975837cd-aff1-404b-a486-d8856ce8ef11-istiod-ca-cert\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820482    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-certs\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-socket\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820516    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-envoy\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-istio-token\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820574    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-credential-socket\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-data\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwx4q\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-kube-api-access-rwx4q\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820995 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-certs\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.820995 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.820949    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-credential-socket\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.821077 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.821059    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-data\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.821193 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.821173    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-socket\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.821244 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.821229    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975837cd-aff1-404b-a486-d8856ce8ef11-istiod-ca-cert\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.822864 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.822844    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-envoy\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.822997 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.822980    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975837cd-aff1-404b-a486-d8856ce8ef11-istio-podinfo\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.833251 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.833223    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwx4q\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-kube-api-access-rwx4q\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.834681 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.834662    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-istio-token\") pod \"e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") " pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.973815 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.973739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:54.994151 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:54.994118    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7509-45945f" podUID="c336f852-881b-4cf2-9f90-4f499551b5cf" containerName="test-curl-7509-45945f" containerID="cri-o://40ce13e9c825eec133604d7946bb046ebf3082e01ca84126b03f8b7e47f8cb1b" gracePeriod=2
Sep 11 19:46:55.146524 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:55.146493    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-85bba4"
Sep 11 19:46:55.224326 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:55.224257    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj2wj\" (UniqueName: \"kubernetes.io/projected/c648c2ad-0204-47af-a679-56ecaf110eae-kube-api-access-mj2wj\") pod \"c648c2ad-0204-47af-a679-56ecaf110eae\" (UID: \"c648c2ad-0204-47af-a679-56ecaf110eae\") "
Sep 11 19:46:55.226378 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:55.226344    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c648c2ad-0204-47af-a679-56ecaf110eae-kube-api-access-mj2wj" (OuterVolumeSpecName: "kube-api-access-mj2wj") pod "c648c2ad-0204-47af-a679-56ecaf110eae" (UID: "c648c2ad-0204-47af-a679-56ecaf110eae"). InnerVolumeSpecName "kube-api-access-mj2wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:46:55.251727 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:55.251708    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"]
Sep 11 19:46:55.253515 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:46:55.253490    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975837cd_aff1_404b_a486_d8856ce8ef11.slice/crio-7b44050dae566cc95feecd34d67cd0d9aa2c432fc85aa698983e0e93c278f7e7 WatchSource:0}: Error finding container 7b44050dae566cc95feecd34d67cd0d9aa2c432fc85aa698983e0e93c278f7e7: Status 404 returned error can't find the container with id 7b44050dae566cc95feecd34d67cd0d9aa2c432fc85aa698983e0e93c278f7e7
Sep 11 19:46:55.325042 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:55.325006    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2wj\" (UniqueName: \"kubernetes.io/projected/c648c2ad-0204-47af-a679-56ecaf110eae-kube-api-access-mj2wj\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:46:55.872247 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:55.872216    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c648c2ad-0204-47af-a679-56ecaf110eae" path="/var/lib/kubelet/pods/c648c2ad-0204-47af-a679-56ecaf110eae/volumes"
Sep 11 19:46:56.005799 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.005760    2571 generic.go:356] "Generic (PLEG): container finished" podID="c648c2ad-0204-47af-a679-56ecaf110eae" containerID="b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13" exitCode=137
Sep 11 19:46:56.006243 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.005847    2571 scope.go:117] "RemoveContainer" containerID="b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13"
Sep 11 19:46:56.006243 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.005884    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-85bba4"
Sep 11 19:46:56.007286 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.007215    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" event={"ID":"975837cd-aff1-404b-a486-d8856ce8ef11","Type":"ContainerStarted","Data":"7b44050dae566cc95feecd34d67cd0d9aa2c432fc85aa698983e0e93c278f7e7"}
Sep 11 19:46:56.014270 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.014251    2571 scope.go:117] "RemoveContainer" containerID="b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13"
Sep 11 19:46:56.014540 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:46:56.014522    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13\": container with ID starting with b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13 not found: ID does not exist" containerID="b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13"
Sep 11 19:46:56.014599 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.014550    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13"} err="failed to get container status \"b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13\": rpc error: code = NotFound desc = could not find container \"b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13\": container with ID starting with b00add879f1b8ddaaedc22e41715f4e51a1bfd712f726b868dbca01d411a5f13 not found: ID does not exist"
Sep 11 19:46:56.843882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.843846    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"]
Sep 11 19:46:56.847179 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.847158    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:46:56.850271 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.850249    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4b032e\""
Sep 11 19:46:56.850557 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.850540    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4b032e-dockercfg-b2qrb\""
Sep 11 19:46:56.861767 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.861741    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"]
Sep 11 19:46:56.940161 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.940127    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296f502b-f979-4b32-a645-55cf4c8fd02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8\" (UID: \"296f502b-f979-4b32-a645-55cf4c8fd02a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:46:56.940325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:56.940173    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwvk\" (UniqueName: \"kubernetes.io/projected/296f502b-f979-4b32-a645-55cf4c8fd02a-kube-api-access-rtwvk\") pod \"payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8\" (UID: \"296f502b-f979-4b32-a645-55cf4c8fd02a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:46:57.017288 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.017260    2571 generic.go:356] "Generic (PLEG): container finished" podID="c336f852-881b-4cf2-9f90-4f499551b5cf" containerID="40ce13e9c825eec133604d7946bb046ebf3082e01ca84126b03f8b7e47f8cb1b" exitCode=137
Sep 11 19:46:57.040669 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.040633    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwvk\" (UniqueName: \"kubernetes.io/projected/296f502b-f979-4b32-a645-55cf4c8fd02a-kube-api-access-rtwvk\") pod \"payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8\" (UID: \"296f502b-f979-4b32-a645-55cf4c8fd02a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:46:57.040890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.040763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296f502b-f979-4b32-a645-55cf4c8fd02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8\" (UID: \"296f502b-f979-4b32-a645-55cf4c8fd02a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:46:57.041684 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.041659    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296f502b-f979-4b32-a645-55cf4c8fd02a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8\" (UID: \"296f502b-f979-4b32-a645-55cf4c8fd02a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:46:57.053876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.053848    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwvk\" (UniqueName: \"kubernetes.io/projected/296f502b-f979-4b32-a645-55cf4c8fd02a-kube-api-access-rtwvk\") pod \"payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8\" (UID: \"296f502b-f979-4b32-a645-55cf4c8fd02a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:46:57.153637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.153606    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-45945f"
Sep 11 19:46:57.160219 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.160194    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:46:57.241524 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.241498    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfwmq\" (UniqueName: \"kubernetes.io/projected/c336f852-881b-4cf2-9f90-4f499551b5cf-kube-api-access-bfwmq\") pod \"c336f852-881b-4cf2-9f90-4f499551b5cf\" (UID: \"c336f852-881b-4cf2-9f90-4f499551b5cf\") "
Sep 11 19:46:57.243421 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.243388    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c336f852-881b-4cf2-9f90-4f499551b5cf-kube-api-access-bfwmq" (OuterVolumeSpecName: "kube-api-access-bfwmq") pod "c336f852-881b-4cf2-9f90-4f499551b5cf" (UID: "c336f852-881b-4cf2-9f90-4f499551b5cf"). InnerVolumeSpecName "kube-api-access-bfwmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:46:57.342628 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.342597    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwmq\" (UniqueName: \"kubernetes.io/projected/c336f852-881b-4cf2-9f90-4f499551b5cf-kube-api-access-bfwmq\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:46:57.441593 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.441567    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"]
Sep 11 19:46:57.444053 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:46:57.444023    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296f502b_f979_4b32_a645_55cf4c8fd02a.slice/crio-eba376de1206c1635e471f9b95f1aa663488e499e4d26751b2f8855b34bdab18 WatchSource:0}: Error finding container eba376de1206c1635e471f9b95f1aa663488e499e4d26751b2f8855b34bdab18: Status 404 returned error can't find the container with id eba376de1206c1635e471f9b95f1aa663488e499e4d26751b2f8855b34bdab18
Sep 11 19:46:57.873726 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:57.873686    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c336f852-881b-4cf2-9f90-4f499551b5cf" path="/var/lib/kubelet/pods/c336f852-881b-4cf2-9f90-4f499551b5cf/volumes"
Sep 11 19:46:58.026330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.026264    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7509-45945f"
Sep 11 19:46:58.026330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.026306    2571 scope.go:117] "RemoveContainer" containerID="40ce13e9c825eec133604d7946bb046ebf3082e01ca84126b03f8b7e47f8cb1b"
Sep 11 19:46:58.028648 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.028610    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8" event={"ID":"296f502b-f979-4b32-a645-55cf4c8fd02a","Type":"ContainerStarted","Data":"eba376de1206c1635e471f9b95f1aa663488e499e4d26751b2f8855b34bdab18"}
Sep 11 19:46:58.308880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.308812    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"]
Sep 11 19:46:58.316687 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.316663    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.320384 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.320361    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6fddae31-openshift-default-dockercfg-gbl2t\""
Sep 11 19:46:58.333476 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.333452    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"]
Sep 11 19:46:58.451045 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451009    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-podinfo\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.451226 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451060    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-envoy\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.451226 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451157    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-credential-socket\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.451226 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-data\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.451372 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4512eb-cd18-4466-8f65-b31196f37d41-istiod-ca-cert\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.451372 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451258    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-certs\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.451372 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451300    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-socket\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.451372 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451324    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-token\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.451372 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.451344    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqz4\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-kube-api-access-bvqz4\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.527452 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.527416    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:46:58.527539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.527495    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:46:58.527539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.527526    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:46:58.552238 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-certs\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552339 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-socket\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552339 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-token\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552339 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552321    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqz4\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-kube-api-access-bvqz4\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552517 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552370    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-podinfo\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552517 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-envoy\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552517 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-credential-socket\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552517 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552510    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-data\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552535    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4512eb-cd18-4466-8f65-b31196f37d41-istiod-ca-cert\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552614    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-certs\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.552990 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552938    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-socket\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.553080 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.552992    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-credential-socket\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.553080 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.553019    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-data\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.553247 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.553224    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4512eb-cd18-4466-8f65-b31196f37d41-istiod-ca-cert\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.555150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.555128    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-podinfo\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.555397 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.555376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-envoy\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.564794 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.564758    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-token\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.565734 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.565713    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqz4\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-kube-api-access-bvqz4\") pod \"e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") " pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.627772 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.627744    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:58.943977 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.943950    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"]
Sep 11 19:46:58.946167 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:46:58.946087    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4512eb_cd18_4466_8f65_b31196f37d41.slice/crio-c968eb4786eb991480c53d744ff563ebaf4edf0793cc71c936771037474ee3e5 WatchSource:0}: Error finding container c968eb4786eb991480c53d744ff563ebaf4edf0793cc71c936771037474ee3e5: Status 404 returned error can't find the container with id c968eb4786eb991480c53d744ff563ebaf4edf0793cc71c936771037474ee3e5
Sep 11 19:46:58.950088 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.950053    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:46:58.950190 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.950167    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:46:58.950269 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:58.950213    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:46:59.038479 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:59.038140    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" event={"ID":"975837cd-aff1-404b-a486-d8856ce8ef11","Type":"ContainerStarted","Data":"f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd"}
Sep 11 19:46:59.040317 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:59.040286    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" event={"ID":"1b4512eb-cd18-4466-8f65-b31196f37d41","Type":"ContainerStarted","Data":"2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb"}
Sep 11 19:46:59.040429 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:59.040346    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" event={"ID":"1b4512eb-cd18-4466-8f65-b31196f37d41","Type":"ContainerStarted","Data":"c968eb4786eb991480c53d744ff563ebaf4edf0793cc71c936771037474ee3e5"}
Sep 11 19:46:59.063482 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:59.063432    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" podStartSLOduration=1.79202873 podStartE2EDuration="5.063416061s" totalImagesPullingTime="3.271387331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:59.060628701 +0000 UTC m=+1055.736565609" watchObservedRunningTime="2026-09-11 19:46:59.063416061 +0000 UTC m=+1055.739352969"
Sep 11 19:46:59.093366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:59.093305    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" podStartSLOduration=1.093287267 podStartE2EDuration="1.093287267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:59.090773935 +0000 UTC m=+1055.766710844" watchObservedRunningTime="2026-09-11 19:46:59.093287267 +0000 UTC m=+1055.769224174"
Sep 11 19:46:59.628347 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:59.628311    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:46:59.974346 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:59.974307    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:46:59.979735 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:46:59.979709    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:47:00.047116 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:00.047060    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8" event={"ID":"296f502b-f979-4b32-a645-55cf4c8fd02a","Type":"ContainerStarted","Data":"950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875"}
Sep 11 19:47:00.047635 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:00.047563    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:47:00.048255 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:00.048235    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:47:00.065078 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:00.065035    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8" podStartSLOduration=2.06164338 podStartE2EDuration="4.065020254s" totalImagesPullingTime="2.003376874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:00.06355982 +0000 UTC m=+1056.739496727" watchObservedRunningTime="2026-09-11 19:47:00.065020254 +0000 UTC m=+1056.740957156"
Sep 11 19:47:00.632987 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:00.632955    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:01.053527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:01.053437    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:47:01.053527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:01.053515    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:01.054024 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:01.054007    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:02.195773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:02.195733    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:47:06.230471 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.230437    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"]
Sep 11 19:47:06.233886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.233868    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:06.237269 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.237253    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7df2ac\""
Sep 11 19:47:06.237347 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.237266    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7df2ac-dockercfg-t777s\""
Sep 11 19:47:06.247173 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.247148    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"]
Sep 11 19:47:06.319740 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.319706    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9k4\" (UniqueName: \"kubernetes.io/projected/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-kube-api-access-nf9k4\") pod \"payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb\" (UID: \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\") " pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:06.319900 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.319771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-plugins-config-volume\") pod \"payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb\" (UID: \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\") " pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:06.421144 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.421096    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9k4\" (UniqueName: \"kubernetes.io/projected/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-kube-api-access-nf9k4\") pod \"payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb\" (UID: \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\") " pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:06.421272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.421183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-plugins-config-volume\") pod \"payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb\" (UID: \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\") " pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:06.421807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.421788    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-plugins-config-volume\") pod \"payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb\" (UID: \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\") " pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:06.430203 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.430173    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9k4\" (UniqueName: \"kubernetes.io/projected/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-kube-api-access-nf9k4\") pod \"payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb\" (UID: \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\") " pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:06.543742 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.543673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:06.823585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:06.823504    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"]
Sep 11 19:47:06.826590 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:47:06.826565    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb06f2c3_d3ed_4805_b9b2_04c9d1066bf3.slice/crio-d1865ef88bba95cdd9e7b33ef8ff075293d401cb8daaed9fbfce9add2cc2c39a WatchSource:0}: Error finding container d1865ef88bba95cdd9e7b33ef8ff075293d401cb8daaed9fbfce9add2cc2c39a: Status 404 returned error can't find the container with id d1865ef88bba95cdd9e7b33ef8ff075293d401cb8daaed9fbfce9add2cc2c39a
Sep 11 19:47:07.087765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.087677    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb" event={"ID":"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3","Type":"ContainerStarted","Data":"ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c"}
Sep 11 19:47:07.087765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.087743    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb" event={"ID":"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3","Type":"ContainerStarted","Data":"d1865ef88bba95cdd9e7b33ef8ff075293d401cb8daaed9fbfce9add2cc2c39a"}
Sep 11 19:47:07.087979 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.087854    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:07.131637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.131588    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb" podStartSLOduration=1.131572011 podStartE2EDuration="1.131572011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:07.131217486 +0000 UTC m=+1063.807154395" watchObservedRunningTime="2026-09-11 19:47:07.131572011 +0000 UTC m=+1063.807508979"
Sep 11 19:47:07.726511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.726474    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"]
Sep 11 19:47:07.729716 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.729698    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:07.732798 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.732781    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6fddae31-dockercfg-vvmnr\""
Sep 11 19:47:07.733188 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.733172    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6fddae31\""
Sep 11 19:47:07.745168 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.745147    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"]
Sep 11 19:47:07.832509 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.832480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9162868-c361-404c-b79e-9a28118283e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj\" (UID: \"c9162868-c361-404c-b79e-9a28118283e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:07.832708 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.832520    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzg52\" (UniqueName: \"kubernetes.io/projected/c9162868-c361-404c-b79e-9a28118283e4-kube-api-access-tzg52\") pod \"payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj\" (UID: \"c9162868-c361-404c-b79e-9a28118283e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:07.932939 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.932906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9162868-c361-404c-b79e-9a28118283e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj\" (UID: \"c9162868-c361-404c-b79e-9a28118283e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:07.933081 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.932958    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzg52\" (UniqueName: \"kubernetes.io/projected/c9162868-c361-404c-b79e-9a28118283e4-kube-api-access-tzg52\") pod \"payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj\" (UID: \"c9162868-c361-404c-b79e-9a28118283e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:07.933608 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.933586    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9162868-c361-404c-b79e-9a28118283e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj\" (UID: \"c9162868-c361-404c-b79e-9a28118283e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:07.943860 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:07.943831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzg52\" (UniqueName: \"kubernetes.io/projected/c9162868-c361-404c-b79e-9a28118283e4-kube-api-access-tzg52\") pod \"payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj\" (UID: \"c9162868-c361-404c-b79e-9a28118283e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:08.040488 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:08.040418    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:08.336446 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:08.336421    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"]
Sep 11 19:47:08.338594 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:47:08.338550    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9162868_c361_404c_b79e_9a28118283e4.slice/crio-df34e986268760ca8423b02cb1e17941c8f76d5b7aaf229796ecda802867cd8b WatchSource:0}: Error finding container df34e986268760ca8423b02cb1e17941c8f76d5b7aaf229796ecda802867cd8b: Status 404 returned error can't find the container with id df34e986268760ca8423b02cb1e17941c8f76d5b7aaf229796ecda802867cd8b
Sep 11 19:47:09.100881 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:09.100843    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj" event={"ID":"c9162868-c361-404c-b79e-9a28118283e4","Type":"ContainerStarted","Data":"1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0"}
Sep 11 19:47:09.101306 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:09.100893    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj" event={"ID":"c9162868-c361-404c-b79e-9a28118283e4","Type":"ContainerStarted","Data":"df34e986268760ca8423b02cb1e17941c8f76d5b7aaf229796ecda802867cd8b"}
Sep 11 19:47:09.101306 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:09.100963    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:09.128899 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:09.128817    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj" podStartSLOduration=2.12880418 podStartE2EDuration="2.12880418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:09.126890807 +0000 UTC m=+1065.802827726" watchObservedRunningTime="2026-09-11 19:47:09.12880418 +0000 UTC m=+1065.804741145"
Sep 11 19:47:12.060287 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.060254    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 19:47:12.691822 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.691787    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"]
Sep 11 19:47:12.695729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.695705    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.700277 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.700250    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 19:47:12.701409 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.701390    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bkww\""
Sep 11 19:47:12.701511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.701408    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-04331c-serving-cert\""
Sep 11 19:47:12.701511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.701419    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 19:47:12.701511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.701419    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:47:12.707507 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.707482    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"]
Sep 11 19:47:12.876379 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.876345    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-metrics-certs\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.876530 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.876390    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.876530 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.876429    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.876530 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.876452    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k49vf\" (UniqueName: \"kubernetes.io/projected/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-kube-api-access-k49vf\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.977513 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.977415    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-metrics-certs\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.977513 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.977473    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.977513 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.977509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.977742 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.977557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k49vf\" (UniqueName: \"kubernetes.io/projected/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-kube-api-access-k49vf\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.977742 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:47:12.977646    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-serving-cert: secret "maas-api-e2e-inf-b-04331c-serving-cert" not found
Sep 11 19:47:12.977742 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:47:12.977714    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls podName:4c876d2f-b8d7-461a-8b57-ca2af6bb89ee nodeName:}" failed. No retries permitted until 2026-09-11 19:47:13.477697944 +0000 UTC m=+1070.153634830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls") pod "maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p" (UID: "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee") : secret "maas-api-e2e-inf-b-04331c-serving-cert" not found
Sep 11 19:47:12.978194 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.978175    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.979931 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.979913    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-metrics-certs\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:12.990618 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:12.990589    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k49vf\" (UniqueName: \"kubernetes.io/projected/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-kube-api-access-k49vf\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:13.483615 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:13.483573    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:13.486089 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:13.486065    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:13.608575 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:13.608538    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:13.889947 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:13.889925    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"]
Sep 11 19:47:13.891720 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:47:13.891687    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c876d2f_b8d7_461a_8b57_ca2af6bb89ee.slice/crio-c47e59b302f4dcf3f4783a3123c62bf9e90c6b03f88118fabd37ccac3544e75c WatchSource:0}: Error finding container c47e59b302f4dcf3f4783a3123c62bf9e90c6b03f88118fabd37ccac3544e75c: Status 404 returned error can't find the container with id c47e59b302f4dcf3f4783a3123c62bf9e90c6b03f88118fabd37ccac3544e75c
Sep 11 19:47:14.127215 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:14.127182    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p" event={"ID":"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee","Type":"ContainerStarted","Data":"c47e59b302f4dcf3f4783a3123c62bf9e90c6b03f88118fabd37ccac3544e75c"}
Sep 11 19:47:17.152550 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:17.151567    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p" event={"ID":"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee","Type":"ContainerStarted","Data":"8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf"}
Sep 11 19:47:17.152550 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:17.152489    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:17.175307 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:17.175257    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p" podStartSLOduration=2.79329199 podStartE2EDuration="5.175238661s" totalImagesPullingTime="2.381946671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:17.172804839 +0000 UTC m=+1073.848741746" watchObservedRunningTime="2026-09-11 19:47:17.175238661 +0000 UTC m=+1073.851175569"
Sep 11 19:47:18.096867 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:18.096838    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:47:20.108371 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:20.108326    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:24.181791 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:24.181764    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:47:25.887506 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:25.865693    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:47:26.216769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.216734    2571 generic.go:356] "Generic (PLEG): container finished" podID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerID="2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb" exitCode=0
Sep 11 19:47:26.216950 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.216808    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" event={"ID":"1b4512eb-cd18-4466-8f65-b31196f37d41","Type":"ContainerDied","Data":"2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb"}
Sep 11 19:47:26.217287 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.217262    2571 scope.go:117] "RemoveContainer" containerID="2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb"
Sep 11 19:47:26.218539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.218503    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:26.218633 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.218598    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:26.218695 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.218648    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:26.218695 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.218677    2571 generic.go:356] "Generic (PLEG): container finished" podID="975837cd-aff1-404b-a486-d8856ce8ef11" containerID="f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd" exitCode=0
Sep 11 19:47:26.218792 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.218721    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" event={"ID":"975837cd-aff1-404b-a486-d8856ce8ef11","Type":"ContainerDied","Data":"f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd"}
Sep 11 19:47:26.219024 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.219010    2571 scope.go:117] "RemoveContainer" containerID="f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd"
Sep 11 19:47:26.219999 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.219977    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:26.220041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.220031    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:26.220074 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:26.220062    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:27.229307 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:27.229271    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" event={"ID":"975837cd-aff1-404b-a486-d8856ce8ef11","Type":"ContainerStarted","Data":"af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892"}
Sep 11 19:47:27.232140 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:27.232090    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" event={"ID":"1b4512eb-cd18-4466-8f65-b31196f37d41","Type":"ContainerStarted","Data":"992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199"}
Sep 11 19:47:27.628422 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:27.628377    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:27.629876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:27.629834    2571 patch_prober.go:28] interesting pod/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 19:47:27.630011 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:27.629925    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 19:47:27.974509 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:27.974421    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:47:27.975686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:27.975660    2571 patch_prober.go:28] interesting pod/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 11 19:47:27.975797 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:27.975709    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 11 19:47:28.629123 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:28.629055    2571 patch_prober.go:28] interesting pod/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 19:47:28.629721 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:28.629684    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 19:47:28.974969 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:28.974862    2571 patch_prober.go:28] interesting pod/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 11 19:47:28.974969 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:28.974926    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 11 19:47:29.629119 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:29.629055    2571 patch_prober.go:28] interesting pod/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 19:47:29.629323 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:29.629143    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 19:47:29.975084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:29.974993    2571 patch_prober.go:28] interesting pod/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 11 19:47:29.975084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:29.975044    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 11 19:47:30.047253 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:30.047210    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:47:30.629184 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:30.629141    2571 patch_prober.go:28] interesting pod/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 19:47:30.629396 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:30.629211    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 19:47:30.974444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:30.974346    2571 patch_prober.go:28] interesting pod/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 11 19:47:30.974444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:30.974410    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 11 19:47:31.054272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.054228    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:31.583668 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.583635    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"]
Sep 11 19:47:31.585854 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.585838    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.590075 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.590055    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a8ee95\"/\"simulator-w4-a8ee95-kserve-self-signed-certs\""
Sep 11 19:47:31.591197 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.591177    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a8ee95\"/\"kube-root-ca.crt\""
Sep 11 19:47:31.591442 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.591420    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a8ee95\"/\"openshift-service-ca.crt\""
Sep 11 19:47:31.591724 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.591486    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a8ee95\"/\"default-dockercfg-4l56f\""
Sep 11 19:47:31.605091 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.605060    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"]
Sep 11 19:47:31.639125 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.632567    2571 patch_prober.go:28] interesting pod/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 11 19:47:31.639125 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.632630    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 11 19:47:31.660382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.660124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdh65\" (UniqueName: \"kubernetes.io/projected/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-kube-api-access-rdh65\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.660382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.660178    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-model-cache\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.660382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.660216    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-home\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.660382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.660237    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tmp-dir\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.660382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.660274    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-dshm\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.660382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.660289    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tls-certs\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.761601 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.761557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-model-cache\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.761804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.761638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-home\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.761804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.761663    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tmp-dir\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.761804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.761726    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-dshm\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.761804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.761754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tls-certs\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.762087 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.761821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdh65\" (UniqueName: \"kubernetes.io/projected/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-kube-api-access-rdh65\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.762672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.762608    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tmp-dir\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.762672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.762613    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-model-cache\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.762834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.762776    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-home\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.764671 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.764650    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-dshm\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.764762 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.764736    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tls-certs\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.770799 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.770779    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdh65\" (UniqueName: \"kubernetes.io/projected/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-kube-api-access-rdh65\") pod \"simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") " pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.895220 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.895180    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:31.974399 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.974369    2571 patch_prober.go:28] interesting pod/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 11 19:47:31.974614 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:31.974592    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 11 19:47:32.223157 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:47:32.223095    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7b8a9a5_78ec_4f02_8da7_ef4465bbce86.slice/crio-0d0b90b4b7dffff27b6b43a40eb847f09cfb09337b5572c51b07d517517503c4 WatchSource:0}: Error finding container 0d0b90b4b7dffff27b6b43a40eb847f09cfb09337b5572c51b07d517517503c4: Status 404 returned error can't find the container with id 0d0b90b4b7dffff27b6b43a40eb847f09cfb09337b5572c51b07d517517503c4
Sep 11 19:47:32.224973 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:32.224808    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"]
Sep 11 19:47:32.272998 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:32.272964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq" event={"ID":"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86","Type":"ContainerStarted","Data":"0d0b90b4b7dffff27b6b43a40eb847f09cfb09337b5572c51b07d517517503c4"}
Sep 11 19:47:32.668489 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:32.668454    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:32.669964 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:32.669941    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:32.978682 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:32.978601    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:47:32.979440 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:32.979417    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:47:33.281013 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:33.280924    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq" event={"ID":"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86","Type":"ContainerStarted","Data":"c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6"}
Sep 11 19:47:33.302348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:33.302302    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq" podStartSLOduration=2.302286622 podStartE2EDuration="2.302286622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:33.299297353 +0000 UTC m=+1089.975234263" watchObservedRunningTime="2026-09-11 19:47:33.302286622 +0000 UTC m=+1089.978223565"
Sep 11 19:47:41.895472 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:41.895436    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:41.895833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:41.895484    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:41.907465 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:41.907436    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:42.337683 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:42.337604    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 19:47:42.662008 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:42.661974    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:47:47.010350 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.010311    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"]
Sep 11 19:47:47.010815 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.010611    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj" podUID="c9162868-c361-404c-b79e-9a28118283e4" containerName="payload-pre-processing" containerID="cri-o://1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0" gracePeriod=30
Sep 11 19:47:47.246940 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.246917    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:47.285232 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.285164    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9162868-c361-404c-b79e-9a28118283e4-plugins-config-volume\") pod \"c9162868-c361-404c-b79e-9a28118283e4\" (UID: \"c9162868-c361-404c-b79e-9a28118283e4\") "
Sep 11 19:47:47.285232 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.285210    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzg52\" (UniqueName: \"kubernetes.io/projected/c9162868-c361-404c-b79e-9a28118283e4-kube-api-access-tzg52\") pod \"c9162868-c361-404c-b79e-9a28118283e4\" (UID: \"c9162868-c361-404c-b79e-9a28118283e4\") "
Sep 11 19:47:47.285603 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.285578    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9162868-c361-404c-b79e-9a28118283e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9162868-c361-404c-b79e-9a28118283e4" (UID: "c9162868-c361-404c-b79e-9a28118283e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:47:47.287361 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.287334    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9162868-c361-404c-b79e-9a28118283e4-kube-api-access-tzg52" (OuterVolumeSpecName: "kube-api-access-tzg52") pod "c9162868-c361-404c-b79e-9a28118283e4" (UID: "c9162868-c361-404c-b79e-9a28118283e4"). InnerVolumeSpecName "kube-api-access-tzg52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:47:47.352718 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.352682    2571 generic.go:356] "Generic (PLEG): container finished" podID="c9162868-c361-404c-b79e-9a28118283e4" containerID="1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0" exitCode=0
Sep 11 19:47:47.352718 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.352707    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"
Sep 11 19:47:47.352907 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.352737    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj" event={"ID":"c9162868-c361-404c-b79e-9a28118283e4","Type":"ContainerDied","Data":"1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0"}
Sep 11 19:47:47.352907 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.352802    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj" event={"ID":"c9162868-c361-404c-b79e-9a28118283e4","Type":"ContainerDied","Data":"df34e986268760ca8423b02cb1e17941c8f76d5b7aaf229796ecda802867cd8b"}
Sep 11 19:47:47.352907 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.352827    2571 scope.go:117] "RemoveContainer" containerID="1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0"
Sep 11 19:47:47.361614 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.361593    2571 scope.go:117] "RemoveContainer" containerID="1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0"
Sep 11 19:47:47.361983 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:47:47.361962    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0\": container with ID starting with 1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0 not found: ID does not exist" containerID="1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0"
Sep 11 19:47:47.362070 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.361994    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0"} err="failed to get container status \"1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0\": rpc error: code = NotFound desc = could not find container \"1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0\": container with ID starting with 1b35cfdbcbb732084ee907b0e7a6a4bfdfc1bb5b90607722e72269dae14136a0 not found: ID does not exist"
Sep 11 19:47:47.385465 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.385439    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"]
Sep 11 19:47:47.385628 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.385610    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy" containerID="cri-o://992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199" gracePeriod=30
Sep 11 19:47:47.385812 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.385778    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9162868-c361-404c-b79e-9a28118283e4-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:47.385812 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.385807    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzg52\" (UniqueName: \"kubernetes.io/projected/c9162868-c361-404c-b79e-9a28118283e4-kube-api-access-tzg52\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:47.395501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.395477    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"]
Sep 11 19:47:47.409333 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.409314    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6fddae31-57c65545c8-2cksj"]
Sep 11 19:47:47.671026 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.670988    2571 patch_prober.go:28] interesting pod/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 19:47:47.671188 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.671044    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:47:47.872798 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:47.872765    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9162868-c361-404c-b79e-9a28118283e4" path="/var/lib/kubelet/pods/c9162868-c361-404c-b79e-9a28118283e4/volumes"
Sep 11 19:47:48.293275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.293244    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"]
Sep 11 19:47:48.293646 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.293598    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9162868-c361-404c-b79e-9a28118283e4" containerName="payload-pre-processing"
Sep 11 19:47:48.293646 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.293611    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9162868-c361-404c-b79e-9a28118283e4" containerName="payload-pre-processing"
Sep 11 19:47:48.293714 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.293670    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9162868-c361-404c-b79e-9a28118283e4" containerName="payload-pre-processing"
Sep 11 19:47:48.296472 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.296455    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.299786 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.299764    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c8fc710a-gw-openshift-default-dockercfg-cdjrx\""
Sep 11 19:47:48.318559 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.318538    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"]
Sep 11 19:47:48.396011 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.395971    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-token\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.396223 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.396045    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-credential-socket\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.396223 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.396071    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-podinfo\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.396223 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.396094    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79t4v\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-kube-api-access-79t4v\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.396223 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.396150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-envoy\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.396223 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.396205    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-certs\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.396444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.396255    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-data\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.396444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.396318    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-socket\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.396444 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.396342    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66e6183-fe5c-4796-9907-c000b7a17d08-istiod-ca-cert\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497608 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497569    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-socket\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497608 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66e6183-fe5c-4796-9907-c000b7a17d08-istiod-ca-cert\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497793 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-token\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497793 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-credential-socket\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497793 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497739    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-podinfo\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497793 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497761    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79t4v\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-kube-api-access-79t4v\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497973 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-envoy\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497973 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497889    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-certs\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.497973 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-data\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.498151 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.497992    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-socket\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.498260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.498239    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-credential-socket\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.498341 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.498317    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-data\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.498436 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.498409    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-certs\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.498486 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.498427    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66e6183-fe5c-4796-9907-c000b7a17d08-istiod-ca-cert\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.500069 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.500047    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-envoy\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.500304 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.500285    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-podinfo\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.507805 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.507785    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-token\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.507971 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.507954    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79t4v\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-kube-api-access-79t4v\") pod \"e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") " pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.606352 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.606290    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:48.883652 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.883628    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"]
Sep 11 19:47:48.885230 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:47:48.885205    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66e6183_fe5c_4796_9907_c000b7a17d08.slice/crio-2c1f72329fe6da76919f4531c35f8ed5d6c9f7c55999cbb1d630ffe4a154e927 WatchSource:0}: Error finding container 2c1f72329fe6da76919f4531c35f8ed5d6c9f7c55999cbb1d630ffe4a154e927: Status 404 returned error can't find the container with id 2c1f72329fe6da76919f4531c35f8ed5d6c9f7c55999cbb1d630ffe4a154e927
Sep 11 19:47:48.887488 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.887457    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:48.887564 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.887525    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:48.887564 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:48.887555    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:47:49.366686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:49.366595    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" event={"ID":"a66e6183-fe5c-4796-9907-c000b7a17d08","Type":"ContainerStarted","Data":"9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471"}
Sep 11 19:47:49.366686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:49.366655    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" event={"ID":"a66e6183-fe5c-4796-9907-c000b7a17d08","Type":"ContainerStarted","Data":"2c1f72329fe6da76919f4531c35f8ed5d6c9f7c55999cbb1d630ffe4a154e927"}
Sep 11 19:47:49.405346 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:49.405295    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" podStartSLOduration=1.405279215 podStartE2EDuration="1.405279215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:49.402170352 +0000 UTC m=+1106.078107260" watchObservedRunningTime="2026-09-11 19:47:49.405279215 +0000 UTC m=+1106.081216117"
Sep 11 19:47:49.606914 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:49.606877    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:49.612204 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:49.612181    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:50.371797 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:50.371770    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:50.372782 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:50.372763    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:47:52.632017 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.631989    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:52.733337 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733247    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-credential-socket\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733337 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733291    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-certs\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733337 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733316    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4512eb-cd18-4466-8f65-b31196f37d41-istiod-ca-cert\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733589 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733375    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-socket\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733589 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733399    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvqz4\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-kube-api-access-bvqz4\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733589 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733543    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-token\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733589 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733559    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:52.733801 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733594    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-podinfo\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733801 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733597    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:52.733801 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733639    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-envoy\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733801 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733663    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-data\") pod \"1b4512eb-cd18-4466-8f65-b31196f37d41\" (UID: \"1b4512eb-cd18-4466-8f65-b31196f37d41\") "
Sep 11 19:47:52.733801 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733708    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:52.734030 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.733812    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4512eb-cd18-4466-8f65-b31196f37d41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:47:52.734030 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.734000    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:52.734159 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.734076    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:52.734159 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.734095    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:52.734159 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.734138    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:52.734159 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.734152    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4512eb-cd18-4466-8f65-b31196f37d41-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:52.735687 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.735661    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:47:52.735800 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.735694    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:47:52.735953 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.735939    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:52.735994 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.735979    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-kube-api-access-bvqz4" (OuterVolumeSpecName: "kube-api-access-bvqz4") pod "1b4512eb-cd18-4466-8f65-b31196f37d41" (UID: "1b4512eb-cd18-4466-8f65-b31196f37d41"). InnerVolumeSpecName "kube-api-access-bvqz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:47:52.835310 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.835285    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvqz4\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-kube-api-access-bvqz4\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:52.835310 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.835308    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:52.835310 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.835317    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:52.835493 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.835326    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:52.835493 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:52.835333    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4512eb-cd18-4466-8f65-b31196f37d41-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:47:53.388032 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.388001    2571 generic.go:356] "Generic (PLEG): container finished" podID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerID="992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199" exitCode=0
Sep 11 19:47:53.388252 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.388046    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"
Sep 11 19:47:53.388252 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.388089    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" event={"ID":"1b4512eb-cd18-4466-8f65-b31196f37d41","Type":"ContainerDied","Data":"992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199"}
Sep 11 19:47:53.388252 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.388172    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj" event={"ID":"1b4512eb-cd18-4466-8f65-b31196f37d41","Type":"ContainerDied","Data":"c968eb4786eb991480c53d744ff563ebaf4edf0793cc71c936771037474ee3e5"}
Sep 11 19:47:53.388252 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.388203    2571 scope.go:117] "RemoveContainer" containerID="992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199"
Sep 11 19:47:53.397436 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.397420    2571 scope.go:117] "RemoveContainer" containerID="2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb"
Sep 11 19:47:53.405205 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.405183    2571 scope.go:117] "RemoveContainer" containerID="992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199"
Sep 11 19:47:53.405461 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:47:53.405445    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199\": container with ID starting with 992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199 not found: ID does not exist" containerID="992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199"
Sep 11 19:47:53.405530 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.405467    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199"} err="failed to get container status \"992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199\": rpc error: code = NotFound desc = could not find container \"992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199\": container with ID starting with 992231dbae6c489a781251f2d39432912078ff376441ef15f76bd5dd125cb199 not found: ID does not exist"
Sep 11 19:47:53.405530 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.405484    2571 scope.go:117] "RemoveContainer" containerID="2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb"
Sep 11 19:47:53.405716 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:47:53.405698    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb\": container with ID starting with 2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb not found: ID does not exist" containerID="2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb"
Sep 11 19:47:53.405754 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.405722    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb"} err="failed to get container status \"2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb\": rpc error: code = NotFound desc = could not find container \"2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb\": container with ID starting with 2b00072584a5e02a099f1ca1264879726628e801e6082a1bc972e44fa5661ebb not found: ID does not exist"
Sep 11 19:47:53.412717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.412691    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"]
Sep 11 19:47:53.415292 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.415271    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6fddae31-openshift-default-c8cfdbc8d-hx8pj"]
Sep 11 19:47:53.872905 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:53.872877    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" path="/var/lib/kubelet/pods/1b4512eb-cd18-4466-8f65-b31196f37d41/volumes"
Sep 11 19:47:54.098139 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.098095    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"]
Sep 11 19:47:54.098439 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.098427    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy"
Sep 11 19:47:54.098479 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.098441    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy"
Sep 11 19:47:54.098479 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.098450    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy"
Sep 11 19:47:54.098479 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.098455    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy"
Sep 11 19:47:54.098583 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.098509    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy"
Sep 11 19:47:54.098583 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.098516    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4512eb-cd18-4466-8f65-b31196f37d41" containerName="istio-proxy"
Sep 11 19:47:54.100412 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.100397    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:54.104522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.104500    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c8fc710a\""
Sep 11 19:47:54.104624 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.104540    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c8fc710a-dockercfg-6h7pn\""
Sep 11 19:47:54.114071 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.114045    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"]
Sep 11 19:47:54.123349 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.123284    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"]
Sep 11 19:47:54.126220 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.126199    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:54.138355 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.138330    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"]
Sep 11 19:47:54.248474 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.248440    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw\" (UID: \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:54.248642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.248487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blt4\" (UniqueName: \"kubernetes.io/projected/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-kube-api-access-9blt4\") pod \"payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw\" (UID: \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:54.248642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.248525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh\" (UID: \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:54.248642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.248582    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74m2d\" (UniqueName: \"kubernetes.io/projected/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-kube-api-access-74m2d\") pod \"payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh\" (UID: \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:54.349968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.349937    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blt4\" (UniqueName: \"kubernetes.io/projected/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-kube-api-access-9blt4\") pod \"payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw\" (UID: \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:54.350132 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.349978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh\" (UID: \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:54.350132 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.350114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74m2d\" (UniqueName: \"kubernetes.io/projected/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-kube-api-access-74m2d\") pod \"payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh\" (UID: \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:54.350240 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.350223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw\" (UID: \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:54.350708 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.350692    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh\" (UID: \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:54.350880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.350863    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw\" (UID: \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:54.361676 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.361651    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blt4\" (UniqueName: \"kubernetes.io/projected/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-kube-api-access-9blt4\") pod \"payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw\" (UID: \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:54.361782 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.361742    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74m2d\" (UniqueName: \"kubernetes.io/projected/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-kube-api-access-74m2d\") pod \"payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh\" (UID: \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:54.410474 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.410381    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:54.438946 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.438908    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:54.705064 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.705040    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"]
Sep 11 19:47:54.706971 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:47:54.706946    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bb6433_fcf7_43eb_b029_0d72b8e6d330.slice/crio-6aec0aa69835c56f5e765206d80e93f6f795f5c0ab15e6a4e775adfccfc9516e WatchSource:0}: Error finding container 6aec0aa69835c56f5e765206d80e93f6f795f5c0ab15e6a4e775adfccfc9516e: Status 404 returned error can't find the container with id 6aec0aa69835c56f5e765206d80e93f6f795f5c0ab15e6a4e775adfccfc9516e
Sep 11 19:47:54.726431 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:54.726404    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"]
Sep 11 19:47:54.738982 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:47:54.738950    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b1ddfa_3541_47f4_963a_d7ea02ac99d3.slice/crio-9cdaa8a010cf67a2d6900f787340608928108a6938ddfac835db04e3d1cf5f9e WatchSource:0}: Error finding container 9cdaa8a010cf67a2d6900f787340608928108a6938ddfac835db04e3d1cf5f9e: Status 404 returned error can't find the container with id 9cdaa8a010cf67a2d6900f787340608928108a6938ddfac835db04e3d1cf5f9e
Sep 11 19:47:55.401217 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:55.401162    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw" event={"ID":"f3bb6433-fcf7-43eb-b029-0d72b8e6d330","Type":"ContainerStarted","Data":"0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4"}
Sep 11 19:47:55.401641 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:55.401235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw" event={"ID":"f3bb6433-fcf7-43eb-b029-0d72b8e6d330","Type":"ContainerStarted","Data":"6aec0aa69835c56f5e765206d80e93f6f795f5c0ab15e6a4e775adfccfc9516e"}
Sep 11 19:47:55.401641 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:55.401259    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:47:55.403026 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:55.403001    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh" event={"ID":"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3","Type":"ContainerStarted","Data":"0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd"}
Sep 11 19:47:55.403149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:55.403048    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh" event={"ID":"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3","Type":"ContainerStarted","Data":"9cdaa8a010cf67a2d6900f787340608928108a6938ddfac835db04e3d1cf5f9e"}
Sep 11 19:47:55.403149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:55.403082    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:47:55.421611 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:55.421561    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw" podStartSLOduration=1.421542497 podStartE2EDuration="1.421542497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:55.41852146 +0000 UTC m=+1112.094458367" watchObservedRunningTime="2026-09-11 19:47:55.421542497 +0000 UTC m=+1112.097479410"
Sep 11 19:47:55.441702 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:47:55.441663    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh" podStartSLOduration=1.441650322 podStartE2EDuration="1.441650322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:55.440599439 +0000 UTC m=+1112.116536349" watchObservedRunningTime="2026-09-11 19:47:55.441650322 +0000 UTC m=+1112.117587230"
Sep 11 19:48:00.950429 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:00.950386    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:48:06.411942 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:06.411902    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:48:06.412423 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:06.412227    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:48:11.492119 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:11.492071    2571 generic.go:356] "Generic (PLEG): container finished" podID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerID="9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471" exitCode=0
Sep 11 19:48:11.492714 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:11.492135    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" event={"ID":"a66e6183-fe5c-4796-9907-c000b7a17d08","Type":"ContainerDied","Data":"9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471"}
Sep 11 19:48:11.492714 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:11.492488    2571 scope.go:117] "RemoveContainer" containerID="9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471"
Sep 11 19:48:11.493542 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:11.493517    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:11.493617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:11.493583    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:11.493651 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:11.493618    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:12.499424 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:12.499392    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" event={"ID":"a66e6183-fe5c-4796-9907-c000b7a17d08","Type":"ContainerStarted","Data":"cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd"}
Sep 11 19:48:12.607438 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:12.607401    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:48:12.607821 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:12.607785    2571 patch_prober.go:28] interesting pod/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:48:12.607998 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:12.607975    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 19:48:13.607136 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:13.607074    2571 patch_prober.go:28] interesting pod/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:48:13.607621 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:13.607174    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 19:48:14.607315 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:14.607272    2571 patch_prober.go:28] interesting pod/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:48:14.607816 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:14.607337    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 19:48:15.607274 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:15.607232    2571 patch_prober.go:28] interesting pod/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:48:15.607452 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:15.607306    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 19:48:16.607455 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:16.607420    2571 patch_prober.go:28] interesting pod/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:48:16.608024 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:16.607990    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 19:48:17.701297 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:17.701244    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:48:17.701779 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:17.701669    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:48:17.702944 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:17.702924    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:48:22.864882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:22.864845    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"]
Sep 11 19:48:22.865386 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:22.865123    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh" podUID="b0b1ddfa-3541-47f4-963a-d7ea02ac99d3" containerName="payload-processing" containerID="cri-o://0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd" gracePeriod=30
Sep 11 19:48:22.870938 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:22.870898    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"]
Sep 11 19:48:22.871226 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:22.871197    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw" podUID="f3bb6433-fcf7-43eb-b029-0d72b8e6d330" containerName="payload-pre-processing" containerID="cri-o://0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4" gracePeriod=30
Sep 11 19:48:23.139289 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.139257    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:48:23.142133 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.142041    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:48:23.146338 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.146314    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"]
Sep 11 19:48:23.146571 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.146541    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy" containerID="cri-o://cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd" gracePeriod=30
Sep 11 19:48:23.204173 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.203933    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9blt4\" (UniqueName: \"kubernetes.io/projected/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-kube-api-access-9blt4\") pod \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\" (UID: \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\") "
Sep 11 19:48:23.204345 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.204208    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74m2d\" (UniqueName: \"kubernetes.io/projected/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-kube-api-access-74m2d\") pod \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\" (UID: \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\") "
Sep 11 19:48:23.204345 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.204249    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-plugins-config-volume\") pod \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\" (UID: \"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3\") "
Sep 11 19:48:23.204478 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.204428    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-plugins-config-volume\") pod \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\" (UID: \"f3bb6433-fcf7-43eb-b029-0d72b8e6d330\") "
Sep 11 19:48:23.204644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.204623    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0b1ddfa-3541-47f4-963a-d7ea02ac99d3" (UID: "b0b1ddfa-3541-47f4-963a-d7ea02ac99d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:48:23.204737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.204720    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:23.204862 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.204835    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3bb6433-fcf7-43eb-b029-0d72b8e6d330" (UID: "f3bb6433-fcf7-43eb-b029-0d72b8e6d330"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:48:23.206187 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.206164    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-kube-api-access-9blt4" (OuterVolumeSpecName: "kube-api-access-9blt4") pod "f3bb6433-fcf7-43eb-b029-0d72b8e6d330" (UID: "f3bb6433-fcf7-43eb-b029-0d72b8e6d330"). InnerVolumeSpecName "kube-api-access-9blt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:48:23.206331 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.206315    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-kube-api-access-74m2d" (OuterVolumeSpecName: "kube-api-access-74m2d") pod "b0b1ddfa-3541-47f4-963a-d7ea02ac99d3" (UID: "b0b1ddfa-3541-47f4-963a-d7ea02ac99d3"). InnerVolumeSpecName "kube-api-access-74m2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:48:23.305568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.305544    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9blt4\" (UniqueName: \"kubernetes.io/projected/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-kube-api-access-9blt4\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:23.305568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.305566    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74m2d\" (UniqueName: \"kubernetes.io/projected/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3-kube-api-access-74m2d\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:23.305568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.305575    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3bb6433-fcf7-43eb-b029-0d72b8e6d330-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:23.571817 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.571778    2571 generic.go:356] "Generic (PLEG): container finished" podID="b0b1ddfa-3541-47f4-963a-d7ea02ac99d3" containerID="0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd" exitCode=0
Sep 11 19:48:23.571990 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.571841    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"
Sep 11 19:48:23.571990 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.571860    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh" event={"ID":"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3","Type":"ContainerDied","Data":"0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd"}
Sep 11 19:48:23.571990 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.571981    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh" event={"ID":"b0b1ddfa-3541-47f4-963a-d7ea02ac99d3","Type":"ContainerDied","Data":"9cdaa8a010cf67a2d6900f787340608928108a6938ddfac835db04e3d1cf5f9e"}
Sep 11 19:48:23.572175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.572018    2571 scope.go:117] "RemoveContainer" containerID="0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd"
Sep 11 19:48:23.573301 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.573282    2571 generic.go:356] "Generic (PLEG): container finished" podID="f3bb6433-fcf7-43eb-b029-0d72b8e6d330" containerID="0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4" exitCode=0
Sep 11 19:48:23.573396 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.573306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw" event={"ID":"f3bb6433-fcf7-43eb-b029-0d72b8e6d330","Type":"ContainerDied","Data":"0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4"}
Sep 11 19:48:23.573396 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.573339    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"
Sep 11 19:48:23.573396 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.573355    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw" event={"ID":"f3bb6433-fcf7-43eb-b029-0d72b8e6d330","Type":"ContainerDied","Data":"6aec0aa69835c56f5e765206d80e93f6f795f5c0ab15e6a4e775adfccfc9516e"}
Sep 11 19:48:23.582388 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.582373    2571 scope.go:117] "RemoveContainer" containerID="0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd"
Sep 11 19:48:23.582674 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:48:23.582655    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd\": container with ID starting with 0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd not found: ID does not exist" containerID="0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd"
Sep 11 19:48:23.582723 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.582686    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd"} err="failed to get container status \"0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd\": rpc error: code = NotFound desc = could not find container \"0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd\": container with ID starting with 0d2ad136625ac16246a329caa8cff7995ad53c1487bf8eca33f812835a015ecd not found: ID does not exist"
Sep 11 19:48:23.582723 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.582710    2571 scope.go:117] "RemoveContainer" containerID="0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4"
Sep 11 19:48:23.590699 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.590682    2571 scope.go:117] "RemoveContainer" containerID="0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4"
Sep 11 19:48:23.590920 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:48:23.590900    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4\": container with ID starting with 0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4 not found: ID does not exist" containerID="0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4"
Sep 11 19:48:23.590983 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.590930    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4"} err="failed to get container status \"0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4\": rpc error: code = NotFound desc = could not find container \"0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4\": container with ID starting with 0e2dd8ecc3b9c37a98f8cf3105dbaa32b76168f5b59631098f4d261983aeb8a4 not found: ID does not exist"
Sep 11 19:48:23.597986 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.597963    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"]
Sep 11 19:48:23.602183 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.602161    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8fc710a-94fc65f58-zmnkh"]
Sep 11 19:48:23.614437 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.614404    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"]
Sep 11 19:48:23.617343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.617322    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c8fc710a-54c998cf78-r4wqw"]
Sep 11 19:48:23.876244 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.876217    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0b1ddfa-3541-47f4-963a-d7ea02ac99d3" path="/var/lib/kubelet/pods/b0b1ddfa-3541-47f4-963a-d7ea02ac99d3/volumes"
Sep 11 19:48:23.876813 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.876732    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3bb6433-fcf7-43eb-b029-0d72b8e6d330" path="/var/lib/kubelet/pods/f3bb6433-fcf7-43eb-b029-0d72b8e6d330/volumes"
Sep 11 19:48:23.885591 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.885573    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"]
Sep 11 19:48:23.885887 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.885874    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b1ddfa-3541-47f4-963a-d7ea02ac99d3" containerName="payload-processing"
Sep 11 19:48:23.885945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.885891    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b1ddfa-3541-47f4-963a-d7ea02ac99d3" containerName="payload-processing"
Sep 11 19:48:23.885945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.885916    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3bb6433-fcf7-43eb-b029-0d72b8e6d330" containerName="payload-pre-processing"
Sep 11 19:48:23.885945 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.885922    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3bb6433-fcf7-43eb-b029-0d72b8e6d330" containerName="payload-pre-processing"
Sep 11 19:48:23.886039 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.885979    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3bb6433-fcf7-43eb-b029-0d72b8e6d330" containerName="payload-pre-processing"
Sep 11 19:48:23.886039 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.885989    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0b1ddfa-3541-47f4-963a-d7ea02ac99d3" containerName="payload-processing"
Sep 11 19:48:23.890389 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.890373    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.893477 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.893459    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5d9a4dfe-openshift-default-dockercfg-bnxqp\""
Sep 11 19:48:23.902222 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.902204    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"]
Sep 11 19:48:23.910403 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-credential-socket\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.910485 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910410    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-socket\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.910485 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-certs\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.910568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910526    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-envoy\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.910568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-podinfo\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.910642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d763154f-2d25-44b5-8bea-eaa1ac354e89-istiod-ca-cert\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.910642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910596    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-data\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.910642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-token\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:23.910738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:23.910648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b66j\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-kube-api-access-8b66j\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.011875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.011847    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-envoy\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.011875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.011879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-podinfo\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.011899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d763154f-2d25-44b5-8bea-eaa1ac354e89-istiod-ca-cert\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.011929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-data\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.011956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-token\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012149 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012052    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b66j\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-kube-api-access-8b66j\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-credential-socket\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012382 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012304    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-socket\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012473 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012378    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-data\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012473 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012396    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-certs\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012571 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012531    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-credential-socket\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012720 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012697    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-socket\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012786 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012735    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-certs\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.012786 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.012754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d763154f-2d25-44b5-8bea-eaa1ac354e89-istiod-ca-cert\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.014730 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.014706    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-envoy\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.014875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.014858    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-podinfo\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.020520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.020498    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-token\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.020876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.020855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b66j\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-kube-api-access-8b66j\") pod \"e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") " pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.200473 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.200405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:24.474872 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.474849    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"]
Sep 11 19:48:24.477146 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:48:24.477110    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd763154f_2d25_44b5_8bea_eaa1ac354e89.slice/crio-0bb62a351b80ac0c229e90e0313cd51df0f82a638e1ac8154bf9d1595a125a6a WatchSource:0}: Error finding container 0bb62a351b80ac0c229e90e0313cd51df0f82a638e1ac8154bf9d1595a125a6a: Status 404 returned error can't find the container with id 0bb62a351b80ac0c229e90e0313cd51df0f82a638e1ac8154bf9d1595a125a6a
Sep 11 19:48:24.479532 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.479500    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:24.479634 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.479562    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:24.479634 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.479593    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:24.581502 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.581458    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" event={"ID":"d763154f-2d25-44b5-8bea-eaa1ac354e89","Type":"ContainerStarted","Data":"1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866"}
Sep 11 19:48:24.581663 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.581528    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" event={"ID":"d763154f-2d25-44b5-8bea-eaa1ac354e89","Type":"ContainerStarted","Data":"0bb62a351b80ac0c229e90e0313cd51df0f82a638e1ac8154bf9d1595a125a6a"}
Sep 11 19:48:24.609168 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:24.609095    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" podStartSLOduration=1.6090749720000002 podStartE2EDuration="1.609074972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:48:24.607301627 +0000 UTC m=+1141.283238536" watchObservedRunningTime="2026-09-11 19:48:24.609074972 +0000 UTC m=+1141.285011881"
Sep 11 19:48:25.201408 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:25.201351    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:25.206249 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:25.206225    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:25.586319 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:25.586238    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:25.587409 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:25.587388    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:28.391507 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.391484    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:48:28.453411 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453339    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-token\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453411 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453373    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-podinfo\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453605 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453416    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-credential-socket\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453605 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453431    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79t4v\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-kube-api-access-79t4v\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453605 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453454    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-data\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453605 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453488    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-envoy\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453605 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453524    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-socket\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453605 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453583    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-certs\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453605 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453605    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66e6183-fe5c-4796-9907-c000b7a17d08-istiod-ca-cert\") pod \"a66e6183-fe5c-4796-9907-c000b7a17d08\" (UID: \"a66e6183-fe5c-4796-9907-c000b7a17d08\") "
Sep 11 19:48:28.453944 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453728    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:48:28.453944 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453818    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:48:28.453944 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453830    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-data" (OuterVolumeSpecName: "istio-data") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:48:28.453944 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453934    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.454230 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453952    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.454230 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.453966    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.454230 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.454003    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:48:28.454230 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.454047    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66e6183-fe5c-4796-9907-c000b7a17d08-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:48:28.455607 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.455581    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-token" (OuterVolumeSpecName: "istio-token") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:48:28.455607 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.455600    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:48:28.455765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.455636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-kube-api-access-79t4v" (OuterVolumeSpecName: "kube-api-access-79t4v") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "kube-api-access-79t4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:48:28.455804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.455772    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a66e6183-fe5c-4796-9907-c000b7a17d08" (UID: "a66e6183-fe5c-4796-9907-c000b7a17d08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:48:28.481275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.481240    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"]
Sep 11 19:48:28.481557 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.481544    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy"
Sep 11 19:48:28.481600 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.481561    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy"
Sep 11 19:48:28.481631 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.481625    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy"
Sep 11 19:48:28.481664 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.481636    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy"
Sep 11 19:48:28.484617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.484601    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:28.487272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.487255    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5d9a4dfe\""
Sep 11 19:48:28.487354 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.487297    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5d9a4dfe-dockercfg-5pjtf\""
Sep 11 19:48:28.498691 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.498670    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"]
Sep 11 19:48:28.522195 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.522175    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"]
Sep 11 19:48:28.522483 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.522469    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy"
Sep 11 19:48:28.522483 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.522485    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerName="istio-proxy"
Sep 11 19:48:28.525741 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.525725    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:28.539171 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.539141    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"]
Sep 11 19:48:28.554312 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554287    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jc5\" (UniqueName: \"kubernetes.io/projected/da2c8979-c24d-4a1b-96c5-d9cf4b443762-kube-api-access-p2jc5\") pod \"payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd\" (UID: \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:28.554412 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzw6x\" (UniqueName: \"kubernetes.io/projected/08b65078-0402-4041-be0e-2a9941ee3ec2-kube-api-access-hzw6x\") pod \"payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n\" (UID: \"08b65078-0402-4041-be0e-2a9941ee3ec2\") " pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:28.554463 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554412    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b65078-0402-4041-be0e-2a9941ee3ec2-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n\" (UID: \"08b65078-0402-4041-be0e-2a9941ee3ec2\") " pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:28.554503 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c8979-c24d-4a1b-96c5-d9cf4b443762-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd\" (UID: \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:28.554540 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554529    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.554572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554540    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.554572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554549    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79t4v\" (UniqueName: \"kubernetes.io/projected/a66e6183-fe5c-4796-9907-c000b7a17d08-kube-api-access-79t4v\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.554572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554557    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.554572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554565    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66e6183-fe5c-4796-9907-c000b7a17d08-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.554572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.554572    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66e6183-fe5c-4796-9907-c000b7a17d08-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:28.603689 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.603653    2571 generic.go:356] "Generic (PLEG): container finished" podID="a66e6183-fe5c-4796-9907-c000b7a17d08" containerID="cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd" exitCode=0
Sep 11 19:48:28.603875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.603718    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"
Sep 11 19:48:28.603875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.603716    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" event={"ID":"a66e6183-fe5c-4796-9907-c000b7a17d08","Type":"ContainerDied","Data":"cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd"}
Sep 11 19:48:28.603875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.603844    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw" event={"ID":"a66e6183-fe5c-4796-9907-c000b7a17d08","Type":"ContainerDied","Data":"2c1f72329fe6da76919f4531c35f8ed5d6c9f7c55999cbb1d630ffe4a154e927"}
Sep 11 19:48:28.603875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.603873    2571 scope.go:117] "RemoveContainer" containerID="cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd"
Sep 11 19:48:28.613282 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.613264    2571 scope.go:117] "RemoveContainer" containerID="9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471"
Sep 11 19:48:28.621409 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.621389    2571 scope.go:117] "RemoveContainer" containerID="cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd"
Sep 11 19:48:28.621667 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:48:28.621649    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd\": container with ID starting with cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd not found: ID does not exist" containerID="cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd"
Sep 11 19:48:28.621713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.621675    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd"} err="failed to get container status \"cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd\": rpc error: code = NotFound desc = could not find container \"cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd\": container with ID starting with cd8f5c1a147db2c32dc6bad3c1c879eb1c28de7da28fd7603e9eb59dbb9a21bd not found: ID does not exist"
Sep 11 19:48:28.621713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.621693    2571 scope.go:117] "RemoveContainer" containerID="9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471"
Sep 11 19:48:28.621957 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:48:28.621939    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471\": container with ID starting with 9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471 not found: ID does not exist" containerID="9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471"
Sep 11 19:48:28.622010 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.621962    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471"} err="failed to get container status \"9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471\": rpc error: code = NotFound desc = could not find container \"9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471\": container with ID starting with 9d3ea641f59069d677b5174efc2f11ed2e16e33790a5eda98084ddf4831a2471 not found: ID does not exist"
Sep 11 19:48:28.625556 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.625532    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"]
Sep 11 19:48:28.628585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.628561    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c8fc710a-gw-openshift-default-74b5d6b6f6-bblnw"]
Sep 11 19:48:28.655685 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.655666    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c8979-c24d-4a1b-96c5-d9cf4b443762-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd\" (UID: \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:28.655781 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.655711    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jc5\" (UniqueName: \"kubernetes.io/projected/da2c8979-c24d-4a1b-96c5-d9cf4b443762-kube-api-access-p2jc5\") pod \"payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd\" (UID: \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:28.655781 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.655736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzw6x\" (UniqueName: \"kubernetes.io/projected/08b65078-0402-4041-be0e-2a9941ee3ec2-kube-api-access-hzw6x\") pod \"payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n\" (UID: \"08b65078-0402-4041-be0e-2a9941ee3ec2\") " pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:28.655781 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.655775    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b65078-0402-4041-be0e-2a9941ee3ec2-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n\" (UID: \"08b65078-0402-4041-be0e-2a9941ee3ec2\") " pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:28.656489 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.656470    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c8979-c24d-4a1b-96c5-d9cf4b443762-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd\" (UID: \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:28.656489 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.656483    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b65078-0402-4041-be0e-2a9941ee3ec2-plugins-config-volume\") pod \"payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n\" (UID: \"08b65078-0402-4041-be0e-2a9941ee3ec2\") " pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:28.663664 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.663643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzw6x\" (UniqueName: \"kubernetes.io/projected/08b65078-0402-4041-be0e-2a9941ee3ec2-kube-api-access-hzw6x\") pod \"payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n\" (UID: \"08b65078-0402-4041-be0e-2a9941ee3ec2\") " pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:28.663738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.663647    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jc5\" (UniqueName: \"kubernetes.io/projected/da2c8979-c24d-4a1b-96c5-d9cf4b443762-kube-api-access-p2jc5\") pod \"payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd\" (UID: \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:28.793693 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.793609    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:28.835911 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:28.835873    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:29.067369 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.067342    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"]
Sep 11 19:48:29.068866 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:48:29.068838    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2c8979_c24d_4a1b_96c5_d9cf4b443762.slice/crio-bef8bbef880794bea6686af5f998334cef1373224734bda9476d393193268034 WatchSource:0}: Error finding container bef8bbef880794bea6686af5f998334cef1373224734bda9476d393193268034: Status 404 returned error can't find the container with id bef8bbef880794bea6686af5f998334cef1373224734bda9476d393193268034
Sep 11 19:48:29.114841 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.114821    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"]
Sep 11 19:48:29.118056 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:48:29.118031    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b65078_0402_4041_be0e_2a9941ee3ec2.slice/crio-e96e0e583d500193e29b3addfdeb021edad3849b1259400da00b939d7830bf01 WatchSource:0}: Error finding container e96e0e583d500193e29b3addfdeb021edad3849b1259400da00b939d7830bf01: Status 404 returned error can't find the container with id e96e0e583d500193e29b3addfdeb021edad3849b1259400da00b939d7830bf01
Sep 11 19:48:29.558238 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.558204    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:48:29.610796 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.610761    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd" event={"ID":"da2c8979-c24d-4a1b-96c5-d9cf4b443762","Type":"ContainerStarted","Data":"f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905"}
Sep 11 19:48:29.610974 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.610825    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:29.610974 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.610849    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd" event={"ID":"da2c8979-c24d-4a1b-96c5-d9cf4b443762","Type":"ContainerStarted","Data":"bef8bbef880794bea6686af5f998334cef1373224734bda9476d393193268034"}
Sep 11 19:48:29.612447 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.612422    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n" event={"ID":"08b65078-0402-4041-be0e-2a9941ee3ec2","Type":"ContainerStarted","Data":"09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a"}
Sep 11 19:48:29.612608 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.612466    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:29.612608 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.612484    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n" event={"ID":"08b65078-0402-4041-be0e-2a9941ee3ec2","Type":"ContainerStarted","Data":"e96e0e583d500193e29b3addfdeb021edad3849b1259400da00b939d7830bf01"}
Sep 11 19:48:29.628000 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.627948    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd" podStartSLOduration=1.627928367 podStartE2EDuration="1.627928367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:48:29.62720935 +0000 UTC m=+1146.303146282" watchObservedRunningTime="2026-09-11 19:48:29.627928367 +0000 UTC m=+1146.303865281"
Sep 11 19:48:29.644947 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.644908    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n" podStartSLOduration=1.6448937190000001 podStartE2EDuration="1.644893719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:48:29.64422177 +0000 UTC m=+1146.320158681" watchObservedRunningTime="2026-09-11 19:48:29.644893719 +0000 UTC m=+1146.320830627"
Sep 11 19:48:29.873081 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:29.873050    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a66e6183-fe5c-4796-9907-c000b7a17d08" path="/var/lib/kubelet/pods/a66e6183-fe5c-4796-9907-c000b7a17d08/volumes"
Sep 11 19:48:37.530331 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.530289    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"]
Sep 11 19:48:37.535211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.535188    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.537937 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.537913    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b032e\"/\"kube-root-ca.crt\""
Sep 11 19:48:37.539181 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.539159    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b032e\"/\"simulator-w0-4b032e-kserve-self-signed-certs\""
Sep 11 19:48:37.539181 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.539171    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b032e\"/\"default-dockercfg-t8xhs\""
Sep 11 19:48:37.539323 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.539164    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b032e\"/\"openshift-service-ca.crt\""
Sep 11 19:48:37.544136 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.544117    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"]
Sep 11 19:48:37.625582 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.625557    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q696w\" (UniqueName: \"kubernetes.io/projected/162965eb-da36-444b-9305-b27bb7ac071c-kube-api-access-q696w\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.625711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.625590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-home\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.625711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.625611    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-dshm\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.625711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.625632    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162965eb-da36-444b-9305-b27bb7ac071c-tls-certs\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.625818 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.625731    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-model-cache\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.625818 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.625800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-tmp-dir\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.728025 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.727987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-tmp-dir\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.728226 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.728088    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q696w\" (UniqueName: \"kubernetes.io/projected/162965eb-da36-444b-9305-b27bb7ac071c-kube-api-access-q696w\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.728226 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.728144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-home\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.728226 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.728185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-dshm\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.728370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.728227    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162965eb-da36-444b-9305-b27bb7ac071c-tls-certs\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.728370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.728274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-model-cache\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.728740 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.728712    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-model-cache\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.729236 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.729213    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-tmp-dir\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.729339 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.729270    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-home\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.733741 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.732633    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162965eb-da36-444b-9305-b27bb7ac071c-tls-certs\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.735260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.735239    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-dshm\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.739412 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.739391    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q696w\" (UniqueName: \"kubernetes.io/projected/162965eb-da36-444b-9305-b27bb7ac071c-kube-api-access-q696w\") pod \"simulator-w0-4b032e-kserve-64d4d588c5-nscpm\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") " pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:37.844643 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:37.844574    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:38.117018 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:38.116988    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"]
Sep 11 19:48:38.118013 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:48:38.117986    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod162965eb_da36_444b_9305_b27bb7ac071c.slice/crio-f7f3fb2f86ac1ab5a6c574c89ce426297d0ed4990552688e4fced306fe5cdba7 WatchSource:0}: Error finding container f7f3fb2f86ac1ab5a6c574c89ce426297d0ed4990552688e4fced306fe5cdba7: Status 404 returned error can't find the container with id f7f3fb2f86ac1ab5a6c574c89ce426297d0ed4990552688e4fced306fe5cdba7
Sep 11 19:48:38.661154 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:38.661084    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm" event={"ID":"162965eb-da36-444b-9305-b27bb7ac071c","Type":"ContainerStarted","Data":"8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d"}
Sep 11 19:48:38.661154 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:38.661162    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm" event={"ID":"162965eb-da36-444b-9305-b27bb7ac071c","Type":"ContainerStarted","Data":"f7f3fb2f86ac1ab5a6c574c89ce426297d0ed4990552688e4fced306fe5cdba7"}
Sep 11 19:48:38.681010 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:38.680959    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm" podStartSLOduration=1.680944008 podStartE2EDuration="1.680944008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:48:38.677462021 +0000 UTC m=+1155.353398928" watchObservedRunningTime="2026-09-11 19:48:38.680944008 +0000 UTC m=+1155.356880955"
Sep 11 19:48:40.619771 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:40.619744    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:40.620210 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:40.619941    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:46.252881 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:46.252791    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:48:47.845372 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:47.845336    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:47.845758 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:47.845382    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:47.857766 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:47.857745    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:48.740800 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:48.740770    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 19:48:49.736716 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:49.736679    2571 generic.go:356] "Generic (PLEG): container finished" podID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerID="1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866" exitCode=0
Sep 11 19:48:49.737120 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:49.736753    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" event={"ID":"d763154f-2d25-44b5-8bea-eaa1ac354e89","Type":"ContainerDied","Data":"1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866"}
Sep 11 19:48:49.737212 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:49.737197    2571 scope.go:117] "RemoveContainer" containerID="1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866"
Sep 11 19:48:49.738213 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:49.738180    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:49.738321 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:49.738265    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:49.738321 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:49.738295    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:50.744944 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:50.744911    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" event={"ID":"d763154f-2d25-44b5-8bea-eaa1ac354e89","Type":"ContainerStarted","Data":"63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd"}
Sep 11 19:48:51.201084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:51.201047    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:51.202137 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:51.202095    2571 patch_prober.go:28] interesting pod/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:48:51.202194 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:51.202179    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 19:48:52.201579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:52.201549    2571 patch_prober.go:28] interesting pod/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:48:52.201953 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:52.201603    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 19:48:53.201289 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:53.201255    2571 patch_prober.go:28] interesting pod/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:48:53.201484 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:53.201307    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 19:48:54.201714 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:54.201673    2571 patch_prober.go:28] interesting pod/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:48:54.202153 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:54.201736    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 19:48:55.201336 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:55.201304    2571 patch_prober.go:28] interesting pod/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 11 19:48:55.201514 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:55.201359    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 11 19:48:55.587163 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:55.587057    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:56.205188 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:56.205158    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:56.206186 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:56.206165    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:48:57.241726 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.241696    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"]
Sep 11 19:48:57.242161 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.241891    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n" podUID="08b65078-0402-4041-be0e-2a9941ee3ec2" containerName="payload-processing" containerID="cri-o://09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a" gracePeriod=30
Sep 11 19:48:57.251276 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.251247    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"]
Sep 11 19:48:57.251536 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.251465    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd" podUID="da2c8979-c24d-4a1b-96c5-d9cf4b443762" containerName="payload-pre-processing" containerID="cri-o://f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905" gracePeriod=30
Sep 11 19:48:57.497001 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.496945    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:57.527031 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.526999    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:57.594961 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.594926    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2jc5\" (UniqueName: \"kubernetes.io/projected/da2c8979-c24d-4a1b-96c5-d9cf4b443762-kube-api-access-p2jc5\") pod \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\" (UID: \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\") "
Sep 11 19:48:57.595163 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.594978    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c8979-c24d-4a1b-96c5-d9cf4b443762-plugins-config-volume\") pod \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\" (UID: \"da2c8979-c24d-4a1b-96c5-d9cf4b443762\") "
Sep 11 19:48:57.595163 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.595001    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzw6x\" (UniqueName: \"kubernetes.io/projected/08b65078-0402-4041-be0e-2a9941ee3ec2-kube-api-access-hzw6x\") pod \"08b65078-0402-4041-be0e-2a9941ee3ec2\" (UID: \"08b65078-0402-4041-be0e-2a9941ee3ec2\") "
Sep 11 19:48:57.595163 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.595065    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b65078-0402-4041-be0e-2a9941ee3ec2-plugins-config-volume\") pod \"08b65078-0402-4041-be0e-2a9941ee3ec2\" (UID: \"08b65078-0402-4041-be0e-2a9941ee3ec2\") "
Sep 11 19:48:57.595483 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.595452    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2c8979-c24d-4a1b-96c5-d9cf4b443762-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da2c8979-c24d-4a1b-96c5-d9cf4b443762" (UID: "da2c8979-c24d-4a1b-96c5-d9cf4b443762"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:48:57.595558 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.595484    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b65078-0402-4041-be0e-2a9941ee3ec2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08b65078-0402-4041-be0e-2a9941ee3ec2" (UID: "08b65078-0402-4041-be0e-2a9941ee3ec2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:48:57.597034 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.597013    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2c8979-c24d-4a1b-96c5-d9cf4b443762-kube-api-access-p2jc5" (OuterVolumeSpecName: "kube-api-access-p2jc5") pod "da2c8979-c24d-4a1b-96c5-d9cf4b443762" (UID: "da2c8979-c24d-4a1b-96c5-d9cf4b443762"). InnerVolumeSpecName "kube-api-access-p2jc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:48:57.597128 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.597029    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b65078-0402-4041-be0e-2a9941ee3ec2-kube-api-access-hzw6x" (OuterVolumeSpecName: "kube-api-access-hzw6x") pod "08b65078-0402-4041-be0e-2a9941ee3ec2" (UID: "08b65078-0402-4041-be0e-2a9941ee3ec2"). InnerVolumeSpecName "kube-api-access-hzw6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:48:57.696156 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.696122    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c8979-c24d-4a1b-96c5-d9cf4b443762-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:57.696156 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.696153    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzw6x\" (UniqueName: \"kubernetes.io/projected/08b65078-0402-4041-be0e-2a9941ee3ec2-kube-api-access-hzw6x\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:57.696156 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.696163    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b65078-0402-4041-be0e-2a9941ee3ec2-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:57.696385 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.696173    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jc5\" (UniqueName: \"kubernetes.io/projected/da2c8979-c24d-4a1b-96c5-d9cf4b443762-kube-api-access-p2jc5\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:48:57.781863 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.781779    2571 generic.go:356] "Generic (PLEG): container finished" podID="da2c8979-c24d-4a1b-96c5-d9cf4b443762" containerID="f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905" exitCode=0
Sep 11 19:48:57.781863 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.781832    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"
Sep 11 19:48:57.782048 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.781869    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd" event={"ID":"da2c8979-c24d-4a1b-96c5-d9cf4b443762","Type":"ContainerDied","Data":"f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905"}
Sep 11 19:48:57.782048 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.781929    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd" event={"ID":"da2c8979-c24d-4a1b-96c5-d9cf4b443762","Type":"ContainerDied","Data":"bef8bbef880794bea6686af5f998334cef1373224734bda9476d393193268034"}
Sep 11 19:48:57.782048 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.781959    2571 scope.go:117] "RemoveContainer" containerID="f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905"
Sep 11 19:48:57.783249 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.783226    2571 generic.go:356] "Generic (PLEG): container finished" podID="08b65078-0402-4041-be0e-2a9941ee3ec2" containerID="09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a" exitCode=0
Sep 11 19:48:57.783376 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.783257    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n" event={"ID":"08b65078-0402-4041-be0e-2a9941ee3ec2","Type":"ContainerDied","Data":"09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a"}
Sep 11 19:48:57.783376 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.783278    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"
Sep 11 19:48:57.783376 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.783309    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n" event={"ID":"08b65078-0402-4041-be0e-2a9941ee3ec2","Type":"ContainerDied","Data":"e96e0e583d500193e29b3addfdeb021edad3849b1259400da00b939d7830bf01"}
Sep 11 19:48:57.794407 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.793618    2571 scope.go:117] "RemoveContainer" containerID="f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905"
Sep 11 19:48:57.795564 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:48:57.795537    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905\": container with ID starting with f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905 not found: ID does not exist" containerID="f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905"
Sep 11 19:48:57.795685 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.795569    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905"} err="failed to get container status \"f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905\": rpc error: code = NotFound desc = could not find container \"f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905\": container with ID starting with f15df1f54a9797c59828615e9f469dc172f2a568dbb0b83c94617795351dd905 not found: ID does not exist"
Sep 11 19:48:57.795685 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.795592    2571 scope.go:117] "RemoveContainer" containerID="09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a"
Sep 11 19:48:57.804127 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.804090    2571 scope.go:117] "RemoveContainer" containerID="09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a"
Sep 11 19:48:57.804433 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:48:57.804405    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a\": container with ID starting with 09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a not found: ID does not exist" containerID="09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a"
Sep 11 19:48:57.804514 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.804443    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a"} err="failed to get container status \"09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a\": rpc error: code = NotFound desc = could not find container \"09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a\": container with ID starting with 09293de1b6b709df5c9a1910209f35a7a441ba85b24a4d601782e6db1d913f1a not found: ID does not exist"
Sep 11 19:48:57.810834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.810800    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"]
Sep 11 19:48:57.817398 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.817379    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5d9a4dfe-56cb684d4b-tw8rd"]
Sep 11 19:48:57.828661 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.828637    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"]
Sep 11 19:48:57.832660 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.832642    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5d9a4dfe-b8669597f-g2b2n"]
Sep 11 19:48:57.873187 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.873157    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b65078-0402-4041-be0e-2a9941ee3ec2" path="/var/lib/kubelet/pods/08b65078-0402-4041-be0e-2a9941ee3ec2/volumes"
Sep 11 19:48:57.873583 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:57.873566    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2c8979-c24d-4a1b-96c5-d9cf4b443762" path="/var/lib/kubelet/pods/da2c8979-c24d-4a1b-96c5-d9cf4b443762/volumes"
Sep 11 19:48:58.505172 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:58.505078    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"]
Sep 11 19:48:58.505609 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:58.505298    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy" containerID="cri-o://63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd" gracePeriod=30
Sep 11 19:48:59.323672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.323638    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"]
Sep 11 19:48:59.324010 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.323996    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b65078-0402-4041-be0e-2a9941ee3ec2" containerName="payload-processing"
Sep 11 19:48:59.324053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.324012    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b65078-0402-4041-be0e-2a9941ee3ec2" containerName="payload-processing"
Sep 11 19:48:59.324053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.324035    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2c8979-c24d-4a1b-96c5-d9cf4b443762" containerName="payload-pre-processing"
Sep 11 19:48:59.324053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.324040    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2c8979-c24d-4a1b-96c5-d9cf4b443762" containerName="payload-pre-processing"
Sep 11 19:48:59.324164 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.324093    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2c8979-c24d-4a1b-96c5-d9cf4b443762" containerName="payload-pre-processing"
Sep 11 19:48:59.324164 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.324115    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b65078-0402-4041-be0e-2a9941ee3ec2" containerName="payload-processing"
Sep 11 19:48:59.328554 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.328537    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.334465 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.334442    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6008a04d-openshift-default-dockercfg-t42mv\""
Sep 11 19:48:59.351567 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.351543    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"]
Sep 11 19:48:59.409272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-certs\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.409272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-envoy\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.409485 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-socket\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.409485 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-credential-socket\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.409485 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7m7t\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-kube-api-access-g7m7t\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.409587 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-data\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.409587 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409580    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-podinfo\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.409656 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e080bf3f-a068-45c6-9cf0-27268ed477fa-istiod-ca-cert\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.409656 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.409645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-token\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.510837 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.510806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-certs\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.510837 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.510844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-envoy\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.510866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-socket\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.510884    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-credential-socket\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.510921    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7m7t\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-kube-api-access-g7m7t\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.510963    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-data\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.511001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-podinfo\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.511025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e080bf3f-a068-45c6-9cf0-27268ed477fa-istiod-ca-cert\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.511317    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-certs\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511669 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.511428    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-data\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511669 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.511491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-token\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511669 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.511650    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-socket\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511819 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.511732    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e080bf3f-a068-45c6-9cf0-27268ed477fa-istiod-ca-cert\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.511819 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.511776    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-credential-socket\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.513140 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.513118    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-envoy\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.513587 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.513567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-podinfo\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.519389 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.519366    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-token\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.519475 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.519410    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7m7t\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-kube-api-access-g7m7t\") pod \"e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") " pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.638640 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.638610    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:48:59.921239 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.921212    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"]
Sep 11 19:48:59.923024 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:48:59.922995    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode080bf3f_a068_45c6_9cf0_27268ed477fa.slice/crio-658421cdfd9f3b7cea6a0cb866975b0885fac7a9d926218890a3c49ee759a64c WatchSource:0}: Error finding container 658421cdfd9f3b7cea6a0cb866975b0885fac7a9d926218890a3c49ee759a64c: Status 404 returned error can't find the container with id 658421cdfd9f3b7cea6a0cb866975b0885fac7a9d926218890a3c49ee759a64c
Sep 11 19:48:59.925343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.925311    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:59.925433 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.925374    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:48:59.925433 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:48:59.925401    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:49:00.803052 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:00.803015    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" event={"ID":"e080bf3f-a068-45c6-9cf0-27268ed477fa","Type":"ContainerStarted","Data":"f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a"}
Sep 11 19:49:00.803534 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:00.803070    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" event={"ID":"e080bf3f-a068-45c6-9cf0-27268ed477fa","Type":"ContainerStarted","Data":"658421cdfd9f3b7cea6a0cb866975b0885fac7a9d926218890a3c49ee759a64c"}
Sep 11 19:49:00.824721 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:00.824663    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" podStartSLOduration=1.824648121 podStartE2EDuration="1.824648121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:49:00.822936057 +0000 UTC m=+1177.498872963" watchObservedRunningTime="2026-09-11 19:49:00.824648121 +0000 UTC m=+1177.500585024"
Sep 11 19:49:01.639066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:01.639033    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:01.643733 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:01.643707    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:01.815854 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:01.815823    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:01.816760 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:01.816738    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:03.740685 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.740662    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:49:03.827275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.827208    2571 generic.go:356] "Generic (PLEG): container finished" podID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerID="63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd" exitCode=0
Sep 11 19:49:03.827395 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.827282    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" event={"ID":"d763154f-2d25-44b5-8bea-eaa1ac354e89","Type":"ContainerDied","Data":"63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd"}
Sep 11 19:49:03.827395 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.827306    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"
Sep 11 19:49:03.827395 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.827333    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66" event={"ID":"d763154f-2d25-44b5-8bea-eaa1ac354e89","Type":"ContainerDied","Data":"0bb62a351b80ac0c229e90e0313cd51df0f82a638e1ac8154bf9d1595a125a6a"}
Sep 11 19:49:03.827395 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.827355    2571 scope.go:117] "RemoveContainer" containerID="63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd"
Sep 11 19:49:03.835795 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.835780    2571 scope.go:117] "RemoveContainer" containerID="1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866"
Sep 11 19:49:03.843709 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.843694    2571 scope.go:117] "RemoveContainer" containerID="63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd"
Sep 11 19:49:03.843978 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:49:03.843956    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd\": container with ID starting with 63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd not found: ID does not exist" containerID="63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd"
Sep 11 19:49:03.844026 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.843986    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd"} err="failed to get container status \"63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd\": rpc error: code = NotFound desc = could not find container \"63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd\": container with ID starting with 63cff25ab92da35931b385b902c9d526991617325a8892289c236840440ac5fd not found: ID does not exist"
Sep 11 19:49:03.844026 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.844002    2571 scope.go:117] "RemoveContainer" containerID="1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866"
Sep 11 19:49:03.844278 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:49:03.844260    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866\": container with ID starting with 1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866 not found: ID does not exist" containerID="1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866"
Sep 11 19:49:03.844332 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.844282    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866"} err="failed to get container status \"1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866\": rpc error: code = NotFound desc = could not find container \"1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866\": container with ID starting with 1409346d291bafc1b319864235caa78a3c2db7de76a94eee4aa9b92ba281e866 not found: ID does not exist"
Sep 11 19:49:03.852586 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852565    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-socket\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.852648 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852602    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-data\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.852648 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852639    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-token\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.852723 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852690    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b66j\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-kube-api-access-8b66j\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.852758 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852733    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-podinfo\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.852807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852782    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-certs\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.852912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852800    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:03.852912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852851    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-credential-socket\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.852912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852868    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-data" (OuterVolumeSpecName: "istio-data") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:03.852912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852900    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-envoy\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.853134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.852929    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d763154f-2d25-44b5-8bea-eaa1ac354e89-istiod-ca-cert\") pod \"d763154f-2d25-44b5-8bea-eaa1ac354e89\" (UID: \"d763154f-2d25-44b5-8bea-eaa1ac354e89\") "
Sep 11 19:49:03.853134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.853014    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:03.853134 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.853119    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:03.853295 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.853277    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:03.853336 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.853302    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:03.853336 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.853317    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:03.853336 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.853331    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:03.853509 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.853488    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d763154f-2d25-44b5-8bea-eaa1ac354e89-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:49:03.854917 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.854899    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:03.854917 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.854907    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-kube-api-access-8b66j" (OuterVolumeSpecName: "kube-api-access-8b66j") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "kube-api-access-8b66j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:49:03.855143 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.855117    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:49:03.855249 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.855144    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-token" (OuterVolumeSpecName: "istio-token") pod "d763154f-2d25-44b5-8bea-eaa1ac354e89" (UID: "d763154f-2d25-44b5-8bea-eaa1ac354e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:49:03.954448 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.954416    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b66j\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-kube-api-access-8b66j\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:03.954579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.954450    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:03.954579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.954466    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:03.954579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.954478    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d763154f-2d25-44b5-8bea-eaa1ac354e89-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:03.954579 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:03.954490    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d763154f-2d25-44b5-8bea-eaa1ac354e89-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:04.144453 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:04.144423    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"]
Sep 11 19:49:04.150627 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:04.150604    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5d9a4dfe-openshift-default-7b54fdd7d4-kbv66"]
Sep 11 19:49:05.872347 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:05.872317    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" path="/var/lib/kubelet/pods/d763154f-2d25-44b5-8bea-eaa1ac354e89/volumes"
Sep 11 19:49:06.613882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.613848    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"]
Sep 11 19:49:06.614258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.614242    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy"
Sep 11 19:49:06.614258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.614257    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy"
Sep 11 19:49:06.614401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.614266    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy"
Sep 11 19:49:06.614401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.614272    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy"
Sep 11 19:49:06.614401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.614330    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy"
Sep 11 19:49:06.614401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.614338    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d763154f-2d25-44b5-8bea-eaa1ac354e89" containerName="istio-proxy"
Sep 11 19:49:06.618836 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.618819    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:06.621744 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.621723    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6008a04d-dockercfg-ckj8q\""
Sep 11 19:49:06.621836 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.621723    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6008a04d\""
Sep 11 19:49:06.630553 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.630529    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"]
Sep 11 19:49:06.667677 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.667654    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:49:06.778737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.778690    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342603a0-d266-4f1d-826c-e724b558d3b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-6008a04d-68687774df-nvd4n\" (UID: \"342603a0-d266-4f1d-826c-e724b558d3b6\") " pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:06.778921 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.778816    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbng\" (UniqueName: \"kubernetes.io/projected/342603a0-d266-4f1d-826c-e724b558d3b6-kube-api-access-zcbng\") pod \"payload-processing-e2e-derive-6008a04d-68687774df-nvd4n\" (UID: \"342603a0-d266-4f1d-826c-e724b558d3b6\") " pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:06.879377 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.879348    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbng\" (UniqueName: \"kubernetes.io/projected/342603a0-d266-4f1d-826c-e724b558d3b6-kube-api-access-zcbng\") pod \"payload-processing-e2e-derive-6008a04d-68687774df-nvd4n\" (UID: \"342603a0-d266-4f1d-826c-e724b558d3b6\") " pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:06.879802 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.879424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342603a0-d266-4f1d-826c-e724b558d3b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-6008a04d-68687774df-nvd4n\" (UID: \"342603a0-d266-4f1d-826c-e724b558d3b6\") " pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:06.880161 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.880138    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342603a0-d266-4f1d-826c-e724b558d3b6-plugins-config-volume\") pod \"payload-processing-e2e-derive-6008a04d-68687774df-nvd4n\" (UID: \"342603a0-d266-4f1d-826c-e724b558d3b6\") " pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:06.889718 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.889698    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbng\" (UniqueName: \"kubernetes.io/projected/342603a0-d266-4f1d-826c-e724b558d3b6-kube-api-access-zcbng\") pod \"payload-processing-e2e-derive-6008a04d-68687774df-nvd4n\" (UID: \"342603a0-d266-4f1d-826c-e724b558d3b6\") " pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:06.928600 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:06.928578    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:07.202615 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:07.202585    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"]
Sep 11 19:49:07.203939 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:49:07.203905    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342603a0_d266_4f1d_826c_e724b558d3b6.slice/crio-acfcac4049ef8504395b5dc8a5c55d5a2fa94827c23e0c2475f7cf9ce553e2af WatchSource:0}: Error finding container acfcac4049ef8504395b5dc8a5c55d5a2fa94827c23e0c2475f7cf9ce553e2af: Status 404 returned error can't find the container with id acfcac4049ef8504395b5dc8a5c55d5a2fa94827c23e0c2475f7cf9ce553e2af
Sep 11 19:49:07.851468 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:07.851433    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n" event={"ID":"342603a0-d266-4f1d-826c-e724b558d3b6","Type":"ContainerStarted","Data":"3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b"}
Sep 11 19:49:07.851648 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:07.851490    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:07.851648 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:07.851511    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n" event={"ID":"342603a0-d266-4f1d-826c-e724b558d3b6","Type":"ContainerStarted","Data":"acfcac4049ef8504395b5dc8a5c55d5a2fa94827c23e0c2475f7cf9ce553e2af"}
Sep 11 19:49:07.872454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:07.872400    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n" podStartSLOduration=1.8723820450000002 podStartE2EDuration="1.872382045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:49:07.870678721 +0000 UTC m=+1184.546615628" watchObservedRunningTime="2026-09-11 19:49:07.872382045 +0000 UTC m=+1184.548318951"
Sep 11 19:49:11.641620 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.641569    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"]
Sep 11 19:49:11.645181 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.645158    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.648047 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.648023    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7df2ac\"/\"autores-model-7df2ac-kserve-self-signed-certs\""
Sep 11 19:49:11.650615 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.650593    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7df2ac\"/\"kube-root-ca.crt\""
Sep 11 19:49:11.650615 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.650612    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7df2ac\"/\"openshift-service-ca.crt\""
Sep 11 19:49:11.650783 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.650657    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7df2ac\"/\"default-dockercfg-gf4cp\""
Sep 11 19:49:11.656659 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.656640    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"]
Sep 11 19:49:11.723389 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.723362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-model-cache\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.723508 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.723420    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5v6\" (UniqueName: \"kubernetes.io/projected/7d1ad398-e543-4cd8-bb22-371150ab445f-kube-api-access-gb5v6\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.723558 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.723505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-tmp-dir\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.723594 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.723558    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-home\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.723594 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.723575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-dshm\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.723594 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.723590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1ad398-e543-4cd8-bb22-371150ab445f-tls-certs\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.824652 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.824623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-home\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.824652 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.824655    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-dshm\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.824833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.824673    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1ad398-e543-4cd8-bb22-371150ab445f-tls-certs\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.824833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.824701    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-model-cache\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.824833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.824737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5v6\" (UniqueName: \"kubernetes.io/projected/7d1ad398-e543-4cd8-bb22-371150ab445f-kube-api-access-gb5v6\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.824833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.824793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-tmp-dir\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.825079 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.825029    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-home\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.825163 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.825096    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-tmp-dir\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.825338 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.825319    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-model-cache\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.826926 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.826908    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-dshm\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.827093 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.827075    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1ad398-e543-4cd8-bb22-371150ab445f-tls-certs\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.834015 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.833973    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5v6\" (UniqueName: \"kubernetes.io/projected/7d1ad398-e543-4cd8-bb22-371150ab445f-kube-api-access-gb5v6\") pod \"autores-model-7df2ac-kserve-78986d5c5-7dtqz\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") " pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:11.954053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:11.953978    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:12.224037 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:12.224009    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"]
Sep 11 19:49:12.225904 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:49:12.225869    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d1ad398_e543_4cd8_bb22_371150ab445f.slice/crio-85ada3934f433aa77ad4a8c1324d5231f5304d19f4a2119d61ac5b88052f2685 WatchSource:0}: Error finding container 85ada3934f433aa77ad4a8c1324d5231f5304d19f4a2119d61ac5b88052f2685: Status 404 returned error can't find the container with id 85ada3934f433aa77ad4a8c1324d5231f5304d19f4a2119d61ac5b88052f2685
Sep 11 19:49:12.880828 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:12.880795    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz" event={"ID":"7d1ad398-e543-4cd8-bb22-371150ab445f","Type":"ContainerStarted","Data":"63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d"}
Sep 11 19:49:12.881188 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:12.880835    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz" event={"ID":"7d1ad398-e543-4cd8-bb22-371150ab445f","Type":"ContainerStarted","Data":"85ada3934f433aa77ad4a8c1324d5231f5304d19f4a2119d61ac5b88052f2685"}
Sep 11 19:49:12.898314 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:12.898265    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz" podStartSLOduration=1.898249822 podStartE2EDuration="1.898249822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:49:12.897810104 +0000 UTC m=+1189.573747011" watchObservedRunningTime="2026-09-11 19:49:12.898249822 +0000 UTC m=+1189.574186729"
Sep 11 19:49:18.858949 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:18.858915    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:21.550191 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:21.550109    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:49:21.954168 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:21.954125    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:21.954381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:21.954184    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:21.968973 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:21.968946    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:22.960247 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:22.960212    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:49:24.954188 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:24.954155    2571 generic.go:356] "Generic (PLEG): container finished" podID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerID="f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a" exitCode=0
Sep 11 19:49:24.954690 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:24.954229    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" event={"ID":"e080bf3f-a068-45c6-9cf0-27268ed477fa","Type":"ContainerDied","Data":"f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a"}
Sep 11 19:49:24.954690 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:24.954683    2571 scope.go:117] "RemoveContainer" containerID="f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a"
Sep 11 19:49:24.955813 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:24.955786    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:49:24.955896 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:24.955857    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:49:24.955961 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:24.955892    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:49:25.960789 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:25.960752    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" event={"ID":"e080bf3f-a068-45c6-9cf0-27268ed477fa","Type":"ContainerStarted","Data":"98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1"}
Sep 11 19:49:26.639582 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:26.639545    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:26.639808 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:26.639785    2571 patch_prober.go:28] interesting pod/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 19:49:26.639891 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:26.639853    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 19:49:27.639267 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:27.639231    2571 patch_prober.go:28] interesting pod/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 19:49:27.639672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:27.639286    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 19:49:28.639974 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:28.639939    2571 patch_prober.go:28] interesting pod/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 19:49:28.640457 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:28.639995    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 19:49:29.639537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:29.639500    2571 patch_prober.go:28] interesting pod/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 19:49:29.639725 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:29.639553    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 19:49:30.240933 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.240896    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"]
Sep 11 19:49:30.244339 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.244321    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.248082 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.248049    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a58a49\"/\"kube-root-ca.crt\""
Sep 11 19:49:30.248192 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.248090    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a58a49\"/\"openshift-service-ca.crt\""
Sep 11 19:49:30.248192 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.248057    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a58a49\"/\"test-model-a58a49-kserve-self-signed-certs\""
Sep 11 19:49:30.248268 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.248211    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a58a49\"/\"default-dockercfg-gkrm9\""
Sep 11 19:49:30.253715 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.253694    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"]
Sep 11 19:49:30.271220 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.271194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm9ls\" (UniqueName: \"kubernetes.io/projected/6db85eb3-e8bb-42d3-b923-cdbe18111b57-kube-api-access-xm9ls\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.271325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.271234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-model-cache\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.271325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.271267    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tls-certs\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.271416 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.271333    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tmp-dir\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.271416 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.271366    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-dshm\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.271416 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.271402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-home\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.371792 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.371761    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tls-certs\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.371952 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.371808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tmp-dir\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.371952 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.371842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-dshm\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.371952 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.371879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-home\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.371952 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.371948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm9ls\" (UniqueName: \"kubernetes.io/projected/6db85eb3-e8bb-42d3-b923-cdbe18111b57-kube-api-access-xm9ls\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.372197 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.371980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-model-cache\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.372293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.372265    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tmp-dir\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.372425 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.372325    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-home\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.372425 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.372360    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-model-cache\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.374189 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.374171    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-dshm\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.374312 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.374297    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tls-certs\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.378948 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.378930    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm9ls\" (UniqueName: \"kubernetes.io/projected/6db85eb3-e8bb-42d3-b923-cdbe18111b57-kube-api-access-xm9ls\") pod \"test-model-a58a49-kserve-8d5c55f49-5q8nh\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") " pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.555458 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.555374    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:30.639584 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.639550    2571 patch_prober.go:28] interesting pod/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 11 19:49:30.639751 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.639620    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 11 19:49:30.824388 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.824358    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"]
Sep 11 19:49:30.827177 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:49:30.827146    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6db85eb3_e8bb_42d3_b923_cdbe18111b57.slice/crio-c3e7031f4c2df0a9f7d81521aa804a35c7a27e7794bcddbedfe2770a51143c60 WatchSource:0}: Error finding container c3e7031f4c2df0a9f7d81521aa804a35c7a27e7794bcddbedfe2770a51143c60: Status 404 returned error can't find the container with id c3e7031f4c2df0a9f7d81521aa804a35c7a27e7794bcddbedfe2770a51143c60
Sep 11 19:49:30.989642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.989596    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh" event={"ID":"6db85eb3-e8bb-42d3-b923-cdbe18111b57","Type":"ContainerStarted","Data":"eae95f350d3029ffea9f12ba5293d485e5218334d09b1fde537200947f80ffb6"}
Sep 11 19:49:30.989854 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:30.989652    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh" event={"ID":"6db85eb3-e8bb-42d3-b923-cdbe18111b57","Type":"ContainerStarted","Data":"c3e7031f4c2df0a9f7d81521aa804a35c7a27e7794bcddbedfe2770a51143c60"}
Sep 11 19:49:31.006507 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:31.006024    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh" podStartSLOduration=1.006004281 podStartE2EDuration="1.006004281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:49:31.005553282 +0000 UTC m=+1207.681490190" watchObservedRunningTime="2026-09-11 19:49:31.006004281 +0000 UTC m=+1207.681941188"
Sep 11 19:49:31.643260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:31.643226    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:31.643622 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:31.643536    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:31.644096 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:31.644080    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:37.957799 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:37.957767    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:49:40.555691 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:40.555651    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:40.555691 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:40.555704    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:40.567879 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:40.567852    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:41.056723 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:41.056694    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:49:43.960988 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:43.960948    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"]
Sep 11 19:49:43.961477 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:43.961255    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n" podUID="342603a0-d266-4f1d-826c-e724b558d3b6" containerName="payload-processing" containerID="cri-o://3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b" gracePeriod=30
Sep 11 19:49:44.208687 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.208660    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:44.248220 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.248136    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"]
Sep 11 19:49:44.248402 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.248374    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy" containerID="cri-o://98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1" gracePeriod=30
Sep 11 19:49:44.295996 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.295965    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342603a0-d266-4f1d-826c-e724b558d3b6-plugins-config-volume\") pod \"342603a0-d266-4f1d-826c-e724b558d3b6\" (UID: \"342603a0-d266-4f1d-826c-e724b558d3b6\") "
Sep 11 19:49:44.296205 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.296032    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcbng\" (UniqueName: \"kubernetes.io/projected/342603a0-d266-4f1d-826c-e724b558d3b6-kube-api-access-zcbng\") pod \"342603a0-d266-4f1d-826c-e724b558d3b6\" (UID: \"342603a0-d266-4f1d-826c-e724b558d3b6\") "
Sep 11 19:49:44.296355 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.296332    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342603a0-d266-4f1d-826c-e724b558d3b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "342603a0-d266-4f1d-826c-e724b558d3b6" (UID: "342603a0-d266-4f1d-826c-e724b558d3b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:49:44.298250 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.298222    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342603a0-d266-4f1d-826c-e724b558d3b6-kube-api-access-zcbng" (OuterVolumeSpecName: "kube-api-access-zcbng") pod "342603a0-d266-4f1d-826c-e724b558d3b6" (UID: "342603a0-d266-4f1d-826c-e724b558d3b6"). InnerVolumeSpecName "kube-api-access-zcbng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:49:44.397385 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.397346    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342603a0-d266-4f1d-826c-e724b558d3b6-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:44.397385 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:44.397380    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbng\" (UniqueName: \"kubernetes.io/projected/342603a0-d266-4f1d-826c-e724b558d3b6-kube-api-access-zcbng\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:45.066990 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.066956    2571 generic.go:356] "Generic (PLEG): container finished" podID="342603a0-d266-4f1d-826c-e724b558d3b6" containerID="3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b" exitCode=0
Sep 11 19:49:45.067491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.067000    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"
Sep 11 19:49:45.067491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.067026    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n" event={"ID":"342603a0-d266-4f1d-826c-e724b558d3b6","Type":"ContainerDied","Data":"3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b"}
Sep 11 19:49:45.067491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.067068    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n" event={"ID":"342603a0-d266-4f1d-826c-e724b558d3b6","Type":"ContainerDied","Data":"acfcac4049ef8504395b5dc8a5c55d5a2fa94827c23e0c2475f7cf9ce553e2af"}
Sep 11 19:49:45.067491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.067089    2571 scope.go:117] "RemoveContainer" containerID="3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b"
Sep 11 19:49:45.082678 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.082654    2571 scope.go:117] "RemoveContainer" containerID="3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b"
Sep 11 19:49:45.082993 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:49:45.082969    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b\": container with ID starting with 3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b not found: ID does not exist" containerID="3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b"
Sep 11 19:49:45.083077 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.083006    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b"} err="failed to get container status \"3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b\": rpc error: code = NotFound desc = could not find container \"3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b\": container with ID starting with 3c08c7ba87281589ac379d9bc7e5d55fcc3fb1c554da25f91fd68e4848ce971b not found: ID does not exist"
Sep 11 19:49:45.093417 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.093391    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"]
Sep 11 19:49:45.095589 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.095566    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6008a04d-68687774df-nvd4n"]
Sep 11 19:49:45.873283 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:45.873254    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="342603a0-d266-4f1d-826c-e724b558d3b6" path="/var/lib/kubelet/pods/342603a0-d266-4f1d-826c-e724b558d3b6/volumes"
Sep 11 19:49:46.645622 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:46.645583    2571 patch_prober.go:28] interesting pod/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 19:49:46.645985 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:46.645690    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:49:49.487304 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.487279    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:49.544375 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544283    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-certs\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544375 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544317    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-envoy\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544375 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544335    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-data\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544402    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-podinfo\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544455    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-credential-socket\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544495    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-socket\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544552    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7m7t\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-kube-api-access-g7m7t\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544848 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544659    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-token\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544848 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544592    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:49.544848 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544666    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:49.544848 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544697    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e080bf3f-a068-45c6-9cf0-27268ed477fa-istiod-ca-cert\") pod \"e080bf3f-a068-45c6-9cf0-27268ed477fa\" (UID: \"e080bf3f-a068-45c6-9cf0-27268ed477fa\") "
Sep 11 19:49:49.544848 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.544810    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:49.545087 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.545037    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:49.545087 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.545080    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e080bf3f-a068-45c6-9cf0-27268ed477fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:49:49.545228 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.545191    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e080bf3f-a068-45c6-9cf0-27268ed477fa-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:49.545228 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.545202    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:49.545228 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.545211    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:49.545228 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.545218    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:49.545228 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.545228    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:49.546645 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.546620    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:49:49.546777 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.546763    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:49:49.546941 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.546917    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:49:49.546941 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.546923    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-kube-api-access-g7m7t" (OuterVolumeSpecName: "kube-api-access-g7m7t") pod "e080bf3f-a068-45c6-9cf0-27268ed477fa" (UID: "e080bf3f-a068-45c6-9cf0-27268ed477fa"). InnerVolumeSpecName "kube-api-access-g7m7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:49:49.646500 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.646458    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:49.646500 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.646493    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7m7t\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-kube-api-access-g7m7t\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:49.646500 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.646504    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:49.646752 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:49.646516    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e080bf3f-a068-45c6-9cf0-27268ed477fa-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:50.094882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.094850    2571 generic.go:356] "Generic (PLEG): container finished" podID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerID="98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1" exitCode=0
Sep 11 19:49:50.095040 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.094898    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"
Sep 11 19:49:50.095040 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.094933    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" event={"ID":"e080bf3f-a068-45c6-9cf0-27268ed477fa","Type":"ContainerDied","Data":"98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1"}
Sep 11 19:49:50.095040 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.094988    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc" event={"ID":"e080bf3f-a068-45c6-9cf0-27268ed477fa","Type":"ContainerDied","Data":"658421cdfd9f3b7cea6a0cb866975b0885fac7a9d926218890a3c49ee759a64c"}
Sep 11 19:49:50.095040 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.095011    2571 scope.go:117] "RemoveContainer" containerID="98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1"
Sep 11 19:49:50.104053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.104026    2571 scope.go:117] "RemoveContainer" containerID="f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a"
Sep 11 19:49:50.112358 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.112336    2571 scope.go:117] "RemoveContainer" containerID="98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1"
Sep 11 19:49:50.112641 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:49:50.112622    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1\": container with ID starting with 98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1 not found: ID does not exist" containerID="98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1"
Sep 11 19:49:50.112695 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.112649    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1"} err="failed to get container status \"98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1\": rpc error: code = NotFound desc = could not find container \"98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1\": container with ID starting with 98c2a0312e114bb4b7e9f562f5b63a5884352e9533388d09881fd68a5423e8a1 not found: ID does not exist"
Sep 11 19:49:50.112695 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.112667    2571 scope.go:117] "RemoveContainer" containerID="f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a"
Sep 11 19:49:50.112934 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:49:50.112904    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a\": container with ID starting with f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a not found: ID does not exist" containerID="f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a"
Sep 11 19:49:50.113052 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.112968    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a"} err="failed to get container status \"f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a\": rpc error: code = NotFound desc = could not find container \"f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a\": container with ID starting with f912eaa236e9b66af8c726739927c0c1d985885c648dcc382aca11cbdcd0289a not found: ID does not exist"
Sep 11 19:49:50.114243 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.114222    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"]
Sep 11 19:49:50.117871 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:50.117850    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6008a04d-openshift-default-c697dfdbd-zhrjc"]
Sep 11 19:49:51.872920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:51.872887    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" path="/var/lib/kubelet/pods/e080bf3f-a068-45c6-9cf0-27268ed477fa/volumes"
Sep 11 19:49:54.135487 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:54.135454    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"]
Sep 11 19:49:55.810093 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:55.810060    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"]
Sep 11 19:49:55.810578 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:55.810280    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb" podUID="fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3" containerName="payload-processing" containerID="cri-o://ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c" gracePeriod=30
Sep 11 19:49:56.058400 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.058373    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:49:56.129601 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.129568    2571 generic.go:356] "Generic (PLEG): container finished" podID="fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3" containerID="ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c" exitCode=0
Sep 11 19:49:56.129774 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.129618    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"
Sep 11 19:49:56.129774 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.129644    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb" event={"ID":"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3","Type":"ContainerDied","Data":"ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c"}
Sep 11 19:49:56.129774 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.129696    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb" event={"ID":"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3","Type":"ContainerDied","Data":"d1865ef88bba95cdd9e7b33ef8ff075293d401cb8daaed9fbfce9add2cc2c39a"}
Sep 11 19:49:56.129774 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.129719    2571 scope.go:117] "RemoveContainer" containerID="ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c"
Sep 11 19:49:56.138613 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.138593    2571 scope.go:117] "RemoveContainer" containerID="ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c"
Sep 11 19:49:56.138887 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:49:56.138869    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c\": container with ID starting with ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c not found: ID does not exist" containerID="ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c"
Sep 11 19:49:56.138935 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.138895    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c"} err="failed to get container status \"ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c\": rpc error: code = NotFound desc = could not find container \"ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c\": container with ID starting with ed2b20ec8e2595cd1442b326246f2393b4b16e5f199392fb3e60e5b30be0464c not found: ID does not exist"
Sep 11 19:49:56.205385 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.205357    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf9k4\" (UniqueName: \"kubernetes.io/projected/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-kube-api-access-nf9k4\") pod \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\" (UID: \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\") "
Sep 11 19:49:56.205511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.205407    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-plugins-config-volume\") pod \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\" (UID: \"fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3\") "
Sep 11 19:49:56.205768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.205746    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3" (UID: "fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:49:56.207407 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.207385    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-kube-api-access-nf9k4" (OuterVolumeSpecName: "kube-api-access-nf9k4") pod "fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3" (UID: "fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3"). InnerVolumeSpecName "kube-api-access-nf9k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:49:56.306480 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.306448    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9k4\" (UniqueName: \"kubernetes.io/projected/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-kube-api-access-nf9k4\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:56.306480 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.306475    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:49:56.465342 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.465298    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"]
Sep 11 19:49:56.469109 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:56.469078    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7df2ac-56956d7f67-pggrb"]
Sep 11 19:49:57.873234 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:49:57.873199    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3" path="/var/lib/kubelet/pods/fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3/volumes"
Sep 11 19:50:02.351381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:02.351344    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:50:09.154094 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.154050    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz" podUID="7d1ad398-e543-4cd8-bb22-371150ab445f" containerName="main" containerID="cri-o://63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d" gracePeriod=45
Sep 11 19:50:09.387432 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.387406    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:50:09.511882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.511756    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-model-cache\") pod \"7d1ad398-e543-4cd8-bb22-371150ab445f\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") "
Sep 11 19:50:09.511882 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.511814    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-tmp-dir\") pod \"7d1ad398-e543-4cd8-bb22-371150ab445f\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") "
Sep 11 19:50:09.512886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.512144    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-dshm\") pod \"7d1ad398-e543-4cd8-bb22-371150ab445f\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") "
Sep 11 19:50:09.512886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.512227    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb5v6\" (UniqueName: \"kubernetes.io/projected/7d1ad398-e543-4cd8-bb22-371150ab445f-kube-api-access-gb5v6\") pod \"7d1ad398-e543-4cd8-bb22-371150ab445f\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") "
Sep 11 19:50:09.512886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.512275    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1ad398-e543-4cd8-bb22-371150ab445f-tls-certs\") pod \"7d1ad398-e543-4cd8-bb22-371150ab445f\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") "
Sep 11 19:50:09.512886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.512308    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-home\") pod \"7d1ad398-e543-4cd8-bb22-371150ab445f\" (UID: \"7d1ad398-e543-4cd8-bb22-371150ab445f\") "
Sep 11 19:50:09.513358 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.513191    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-home" (OuterVolumeSpecName: "home") pod "7d1ad398-e543-4cd8-bb22-371150ab445f" (UID: "7d1ad398-e543-4cd8-bb22-371150ab445f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:09.513613 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.513462    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d1ad398-e543-4cd8-bb22-371150ab445f" (UID: "7d1ad398-e543-4cd8-bb22-371150ab445f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:09.513966 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.513710    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d1ad398-e543-4cd8-bb22-371150ab445f" (UID: "7d1ad398-e543-4cd8-bb22-371150ab445f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:09.519075 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.519047    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-dshm" (OuterVolumeSpecName: "dshm") pod "7d1ad398-e543-4cd8-bb22-371150ab445f" (UID: "7d1ad398-e543-4cd8-bb22-371150ab445f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:09.520627 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.520604    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1ad398-e543-4cd8-bb22-371150ab445f-kube-api-access-gb5v6" (OuterVolumeSpecName: "kube-api-access-gb5v6") pod "7d1ad398-e543-4cd8-bb22-371150ab445f" (UID: "7d1ad398-e543-4cd8-bb22-371150ab445f"). InnerVolumeSpecName "kube-api-access-gb5v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:50:09.521034 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.521014    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1ad398-e543-4cd8-bb22-371150ab445f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d1ad398-e543-4cd8-bb22-371150ab445f" (UID: "7d1ad398-e543-4cd8-bb22-371150ab445f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:50:09.613441 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.613419    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:09.613441 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.613440    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:09.613563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.613448    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:09.613563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.613457    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5v6\" (UniqueName: \"kubernetes.io/projected/7d1ad398-e543-4cd8-bb22-371150ab445f-kube-api-access-gb5v6\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:09.613563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.613466    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1ad398-e543-4cd8-bb22-371150ab445f-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:09.613563 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:09.613475    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1ad398-e543-4cd8-bb22-371150ab445f-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:10.199778 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.199698    2571 generic.go:356] "Generic (PLEG): container finished" podID="7d1ad398-e543-4cd8-bb22-371150ab445f" containerID="63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d" exitCode=0
Sep 11 19:50:10.199778 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.199748    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"
Sep 11 19:50:10.200207 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.199752    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz" event={"ID":"7d1ad398-e543-4cd8-bb22-371150ab445f","Type":"ContainerDied","Data":"63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d"}
Sep 11 19:50:10.200207 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.199840    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz" event={"ID":"7d1ad398-e543-4cd8-bb22-371150ab445f","Type":"ContainerDied","Data":"85ada3934f433aa77ad4a8c1324d5231f5304d19f4a2119d61ac5b88052f2685"}
Sep 11 19:50:10.200207 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.199856    2571 scope.go:117] "RemoveContainer" containerID="63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d"
Sep 11 19:50:10.208201 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.208185    2571 scope.go:117] "RemoveContainer" containerID="63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d"
Sep 11 19:50:10.208457 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:50:10.208440    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d\": container with ID starting with 63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d not found: ID does not exist" containerID="63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d"
Sep 11 19:50:10.208506 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.208466    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d"} err="failed to get container status \"63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d\": rpc error: code = NotFound desc = could not find container \"63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d\": container with ID starting with 63756fe3d462b34598a64c04233d81789e0f3cb14014bb8e7da4af242330c45d not found: ID does not exist"
Sep 11 19:50:10.217241 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.217221    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"]
Sep 11 19:50:10.220769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:10.220749    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7df2ac/autores-model-7df2ac-kserve-78986d5c5-7dtqz"]
Sep 11 19:50:11.872785 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:11.872749    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1ad398-e543-4cd8-bb22-371150ab445f" path="/var/lib/kubelet/pods/7d1ad398-e543-4cd8-bb22-371150ab445f/volumes"
Sep 11 19:50:13.654005 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.653969    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"]
Sep 11 19:50:13.654471 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654448    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3" containerName="payload-processing"
Sep 11 19:50:13.654520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654477    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3" containerName="payload-processing"
Sep 11 19:50:13.654520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654495    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy"
Sep 11 19:50:13.654520 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654503    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy"
Sep 11 19:50:13.654617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654522    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="342603a0-d266-4f1d-826c-e724b558d3b6" containerName="payload-processing"
Sep 11 19:50:13.654617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654533    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="342603a0-d266-4f1d-826c-e724b558d3b6" containerName="payload-processing"
Sep 11 19:50:13.654617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654547    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy"
Sep 11 19:50:13.654617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654555    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy"
Sep 11 19:50:13.654617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654575    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1ad398-e543-4cd8-bb22-371150ab445f" containerName="main"
Sep 11 19:50:13.654617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654584    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1ad398-e543-4cd8-bb22-371150ab445f" containerName="main"
Sep 11 19:50:13.654807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654661    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy"
Sep 11 19:50:13.654807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654677    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb06f2c3-d3ed-4805-b9b2-04c9d1066bf3" containerName="payload-processing"
Sep 11 19:50:13.654807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654687    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1ad398-e543-4cd8-bb22-371150ab445f" containerName="main"
Sep 11 19:50:13.654807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.654697    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="342603a0-d266-4f1d-826c-e724b558d3b6" containerName="payload-processing"
Sep 11 19:50:13.659182 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.659164    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.662035 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.662017    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a8ee95\"/\"premium-w4-a8ee95-kserve-self-signed-certs\""
Sep 11 19:50:13.670798 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.670778    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"]
Sep 11 19:50:13.746652 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.746626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-home\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.746756 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.746656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdbj\" (UniqueName: \"kubernetes.io/projected/d20a978b-7298-46f0-aece-b8cd25938cf2-kube-api-access-zsdbj\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.746756 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.746727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-tmp-dir\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.746756 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.746751    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20a978b-7298-46f0-aece-b8cd25938cf2-tls-certs\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.746860 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.746831    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-model-cache\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.746901 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.746886    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-dshm\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848086 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848057    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-dshm\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-home\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848136    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdbj\" (UniqueName: \"kubernetes.io/projected/d20a978b-7298-46f0-aece-b8cd25938cf2-kube-api-access-zsdbj\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-tmp-dir\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848193    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20a978b-7298-46f0-aece-b8cd25938cf2-tls-certs\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848418 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-model-cache\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848559 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848534    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-home\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848671 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848585    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-tmp-dir\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.848671 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.848616    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-model-cache\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.850314 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.850293    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-dshm\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.850398 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.850388    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20a978b-7298-46f0-aece-b8cd25938cf2-tls-certs\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.856749 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.856728    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdbj\" (UniqueName: \"kubernetes.io/projected/d20a978b-7298-46f0-aece-b8cd25938cf2-kube-api-access-zsdbj\") pod \"premium-w4-a8ee95-kserve-75d86f96b7-2pvrx\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") " pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:13.968570 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:13.968508    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:14.238340 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:14.237528    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"]
Sep 11 19:50:14.241056 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:50:14.241029    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd20a978b_7298_46f0_aece_b8cd25938cf2.slice/crio-a756b0cf9cec9eb630a3d9e3a22475f1761ab3616531cce818e42756e10f04c9 WatchSource:0}: Error finding container a756b0cf9cec9eb630a3d9e3a22475f1761ab3616531cce818e42756e10f04c9: Status 404 returned error can't find the container with id a756b0cf9cec9eb630a3d9e3a22475f1761ab3616531cce818e42756e10f04c9
Sep 11 19:50:15.226810 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:15.226771    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx" event={"ID":"d20a978b-7298-46f0-aece-b8cd25938cf2","Type":"ContainerStarted","Data":"35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a"}
Sep 11 19:50:15.227229 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:15.226818    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx" event={"ID":"d20a978b-7298-46f0-aece-b8cd25938cf2","Type":"ContainerStarted","Data":"a756b0cf9cec9eb630a3d9e3a22475f1761ab3616531cce818e42756e10f04c9"}
Sep 11 19:50:15.244119 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:15.244069    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx" podStartSLOduration=2.244054566 podStartE2EDuration="2.244054566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:50:15.242411063 +0000 UTC m=+1251.918347969" watchObservedRunningTime="2026-09-11 19:50:15.244054566 +0000 UTC m=+1251.919991507"
Sep 11 19:50:20.947986 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:20.947954    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"]
Sep 11 19:50:20.948443 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:20.948415    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e080bf3f-a068-45c6-9cf0-27268ed477fa" containerName="istio-proxy"
Sep 11 19:50:20.951500 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:20.951483    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:20.954323 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:20.954298    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b032e\"/\"premium-w0-4b032e-kserve-self-signed-certs\""
Sep 11 19:50:20.962191 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:20.962171    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"]
Sep 11 19:50:21.004854 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.004832    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db71ec-b9b9-4de7-91b7-be46436e8270-tls-certs\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.004974 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.004867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-dshm\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.004974 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.004885    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-model-cache\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.004974 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.004902    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhsp\" (UniqueName: \"kubernetes.io/projected/22db71ec-b9b9-4de7-91b7-be46436e8270-kube-api-access-hqhsp\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.005170 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.005007    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-home\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.005170 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.005120    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-tmp-dir\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106427 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-dshm\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106614 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-model-cache\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106614 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106455    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhsp\" (UniqueName: \"kubernetes.io/projected/22db71ec-b9b9-4de7-91b7-be46436e8270-kube-api-access-hqhsp\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106614 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106481    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-home\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106614 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-tmp-dir\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106614 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db71ec-b9b9-4de7-91b7-be46436e8270-tls-certs\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106872 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-model-cache\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106940 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106919    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-home\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.106978 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.106960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-tmp-dir\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.108577 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.108554    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-dshm\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.108842 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.108827    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db71ec-b9b9-4de7-91b7-be46436e8270-tls-certs\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.116717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.116695    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhsp\" (UniqueName: \"kubernetes.io/projected/22db71ec-b9b9-4de7-91b7-be46436e8270-kube-api-access-hqhsp\") pod \"premium-w0-4b032e-kserve-58dcb89dfd-ncvk6\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") " pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.154319 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.154298    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:50:21.260340 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.260282    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:21.547997 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:21.547974    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"]
Sep 11 19:50:21.550271 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:50:21.550238    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22db71ec_b9b9_4de7_91b7_be46436e8270.slice/crio-809f1af66255624d0437ec396ff11cc45c50100b3d5960c139717ac565f61e95 WatchSource:0}: Error finding container 809f1af66255624d0437ec396ff11cc45c50100b3d5960c139717ac565f61e95: Status 404 returned error can't find the container with id 809f1af66255624d0437ec396ff11cc45c50100b3d5960c139717ac565f61e95
Sep 11 19:50:22.267768 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:22.267733    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6" event={"ID":"22db71ec-b9b9-4de7-91b7-be46436e8270","Type":"ContainerStarted","Data":"ab72ebc5262a2c4809912c1d8b8e14730e4d6088e0d70bdb25e01db1e677610e"}
Sep 11 19:50:22.268143 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:22.267775    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6" event={"ID":"22db71ec-b9b9-4de7-91b7-be46436e8270","Type":"ContainerStarted","Data":"809f1af66255624d0437ec396ff11cc45c50100b3d5960c139717ac565f61e95"}
Sep 11 19:50:22.288544 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:22.288500    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6" podStartSLOduration=2.288485621 podStartE2EDuration="2.288485621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:50:22.287379423 +0000 UTC m=+1258.963316329" watchObservedRunningTime="2026-09-11 19:50:22.288485621 +0000 UTC m=+1258.964422527"
Sep 11 19:50:23.969221 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:23.969195    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:23.969510 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:23.969233    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:23.981251 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:23.981228    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:24.288569 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:24.288544    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 19:50:31.261399 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:31.261366    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:31.261753 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:31.261412    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:31.273052 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:31.273030    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:31.324464 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:31.324443    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 19:50:34.455590 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.455559    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"]
Sep 11 19:50:34.460024 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.460004    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.462746 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.462722    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-04331c\"/\"openshift-service-ca.crt\""
Sep 11 19:50:34.464131 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.464097    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-04331c\"/\"default-dockercfg-k8vl5\""
Sep 11 19:50:34.464229 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.464147    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-04331c\"/\"kube-root-ca.crt\""
Sep 11 19:50:34.464229 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.464200    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-04331c\"/\"test-model-04331c-kserve-self-signed-certs\""
Sep 11 19:50:34.470560 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.470539    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"]
Sep 11 19:50:34.515325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.515302    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-home\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.515424 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.515341    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-dshm\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.515424 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.515363    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-tmp-dir\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.515424 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.515411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvfxm\" (UniqueName: \"kubernetes.io/projected/4d1be39c-c770-4c09-80bb-83094f96345d-kube-api-access-qvfxm\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.515548 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.515428    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-model-cache\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.515548 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.515443    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d1be39c-c770-4c09-80bb-83094f96345d-tls-certs\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.616721 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.616692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-dshm\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.616816 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.616726    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-tmp-dir\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.616816 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.616764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvfxm\" (UniqueName: \"kubernetes.io/projected/4d1be39c-c770-4c09-80bb-83094f96345d-kube-api-access-qvfxm\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.616816 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.616790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-model-cache\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.616962 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.616814    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d1be39c-c770-4c09-80bb-83094f96345d-tls-certs\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.616962 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.616918    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-home\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.617230 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.617210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-tmp-dir\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.617296 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.617276    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-model-cache\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.617296 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.617285    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-home\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.619057 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.619032    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-dshm\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.619348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.619332    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d1be39c-c770-4c09-80bb-83094f96345d-tls-certs\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.625060 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.625040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvfxm\" (UniqueName: \"kubernetes.io/projected/4d1be39c-c770-4c09-80bb-83094f96345d-kube-api-access-qvfxm\") pod \"test-model-04331c-kserve-5759cd96f7-8gxnd\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") " pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:34.769341 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:34.769279    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:35.041182 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:35.041156    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"]
Sep 11 19:50:35.042807 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:50:35.042780    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d1be39c_c770_4c09_80bb_83094f96345d.slice/crio-276ee748ea41feeb5873b248e62fe6f1b0f170bc7dc39bcb10844838e8e43436 WatchSource:0}: Error finding container 276ee748ea41feeb5873b248e62fe6f1b0f170bc7dc39bcb10844838e8e43436: Status 404 returned error can't find the container with id 276ee748ea41feeb5873b248e62fe6f1b0f170bc7dc39bcb10844838e8e43436
Sep 11 19:50:35.335476 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:35.335386    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd" event={"ID":"4d1be39c-c770-4c09-80bb-83094f96345d","Type":"ContainerStarted","Data":"f568fd1625044e5a1ea24e1515d7420d779f032f2385763274eec5d24130f0f0"}
Sep 11 19:50:35.335476 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:35.335438    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd" event={"ID":"4d1be39c-c770-4c09-80bb-83094f96345d","Type":"ContainerStarted","Data":"276ee748ea41feeb5873b248e62fe6f1b0f170bc7dc39bcb10844838e8e43436"}
Sep 11 19:50:35.352840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:35.352797    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd" podStartSLOduration=1.352783243 podStartE2EDuration="1.352783243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:50:35.352128111 +0000 UTC m=+1272.028065027" watchObservedRunningTime="2026-09-11 19:50:35.352783243 +0000 UTC m=+1272.028720150"
Sep 11 19:50:39.093843 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:39.093807    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"]
Sep 11 19:50:39.094253 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:39.094005    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy" containerID="cri-o://af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892" gracePeriod=30
Sep 11 19:50:39.217030 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:39.216996    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:50:44.331855 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.331828    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:50:44.384548 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.384512    2571 generic.go:356] "Generic (PLEG): container finished" podID="975837cd-aff1-404b-a486-d8856ce8ef11" containerID="af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892" exitCode=0
Sep 11 19:50:44.384686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.384555    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"
Sep 11 19:50:44.384686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.384585    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" event={"ID":"975837cd-aff1-404b-a486-d8856ce8ef11","Type":"ContainerDied","Data":"af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892"}
Sep 11 19:50:44.384686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.384637    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5" event={"ID":"975837cd-aff1-404b-a486-d8856ce8ef11","Type":"ContainerDied","Data":"7b44050dae566cc95feecd34d67cd0d9aa2c432fc85aa698983e0e93c278f7e7"}
Sep 11 19:50:44.384686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.384661    2571 scope.go:117] "RemoveContainer" containerID="af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892"
Sep 11 19:50:44.393161 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.393135    2571 scope.go:117] "RemoveContainer" containerID="f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd"
Sep 11 19:50:44.396337 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396318    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwx4q\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-kube-api-access-rwx4q\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396416 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396361    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-certs\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396416 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396381    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-data\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396566 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396461    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-istio-token\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396566 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396488    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-credential-socket\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396566 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396517    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975837cd-aff1-404b-a486-d8856ce8ef11-istiod-ca-cert\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396566 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396557    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-socket\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396615    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-envoy\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396614    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:44.396765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396645    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975837cd-aff1-404b-a486-d8856ce8ef11-istio-podinfo\") pod \"975837cd-aff1-404b-a486-d8856ce8ef11\" (UID: \"975837cd-aff1-404b-a486-d8856ce8ef11\") "
Sep 11 19:50:44.396920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396816    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:44.396920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.396834    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:44.397027 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.397015    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.397081 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.397036    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.397081 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.397051    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.397215 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.397161    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975837cd-aff1-404b-a486-d8856ce8ef11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:50:44.397275 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.397235    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-data" (OuterVolumeSpecName: "istio-data") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:44.398664 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.398636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-kube-api-access-rwx4q" (OuterVolumeSpecName: "kube-api-access-rwx4q") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "kube-api-access-rwx4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:50:44.398755 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.398657    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-istio-token" (OuterVolumeSpecName: "istio-token") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:50:44.398800 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.398780    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:44.399293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.399279    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/975837cd-aff1-404b-a486-d8856ce8ef11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "975837cd-aff1-404b-a486-d8856ce8ef11" (UID: "975837cd-aff1-404b-a486-d8856ce8ef11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:50:44.414092 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.414076    2571 scope.go:117] "RemoveContainer" containerID="af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892"
Sep 11 19:50:44.414378 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:50:44.414360    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892\": container with ID starting with af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892 not found: ID does not exist" containerID="af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892"
Sep 11 19:50:44.414434 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.414386    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892"} err="failed to get container status \"af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892\": rpc error: code = NotFound desc = could not find container \"af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892\": container with ID starting with af7a1f1aa3b7189a74b4fe259b7140e5df661da930aa55fd6571b1e24645b892 not found: ID does not exist"
Sep 11 19:50:44.414434 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.414403    2571 scope.go:117] "RemoveContainer" containerID="f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd"
Sep 11 19:50:44.414672 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:50:44.414658    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd\": container with ID starting with f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd not found: ID does not exist" containerID="f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd"
Sep 11 19:50:44.414713 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.414675    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd"} err="failed to get container status \"f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd\": rpc error: code = NotFound desc = could not find container \"f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd\": container with ID starting with f3e9925ef1051a69b5a64b85db1b594527a0fa7fabd9ab43d59248a156f271cd not found: ID does not exist"
Sep 11 19:50:44.497791 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.497771    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.497791 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.497790    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975837cd-aff1-404b-a486-d8856ce8ef11-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.497913 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.497799    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.497913 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.497807    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975837cd-aff1-404b-a486-d8856ce8ef11-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.497913 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.497815    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwx4q\" (UniqueName: \"kubernetes.io/projected/975837cd-aff1-404b-a486-d8856ce8ef11-kube-api-access-rwx4q\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.497913 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.497823    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975837cd-aff1-404b-a486-d8856ce8ef11-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:50:44.706340 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.706307    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"]
Sep 11 19:50:44.712707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.712686    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7df2ac-openshift-default-5f97546b56-4gkz5"]
Sep 11 19:50:44.770156 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.770135    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:44.770245 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.770172    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:44.782336 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:44.782314    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:45.402407 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:45.402375    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:50:45.873291 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:45.873259    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" path="/var/lib/kubelet/pods/975837cd-aff1-404b-a486-d8856ce8ef11/volumes"
Sep 11 19:50:50.113471 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.113439    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"]
Sep 11 19:50:50.113912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.113890    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy"
Sep 11 19:50:50.113986 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.113916    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy"
Sep 11 19:50:50.113986 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.113932    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy"
Sep 11 19:50:50.113986 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.113940    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy"
Sep 11 19:50:50.114151 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.114022    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy"
Sep 11 19:50:50.118472 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.118450    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.121069 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.121045    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 19:50:50.121425 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.121409    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f76076-openshift-default-dockercfg-6pwz2\""
Sep 11 19:50:50.129757 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.129728    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"]
Sep 11 19:50:50.235515 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235486    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfp8\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-kube-api-access-ksfp8\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.235630 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235521    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-socket\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.235630 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-certs\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.235630 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-envoy\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.235737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-data\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.235737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235654    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-token\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.235737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-credential-socket\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.235737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235686    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-podinfo\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.235737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.235729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istiod-ca-cert\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.336910 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.336854    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-envoy\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.336910 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.336890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-data\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.336914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-token\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.336933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-credential-socket\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.336952    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-podinfo\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.336974    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istiod-ca-cert\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.337005    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfp8\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-kube-api-access-ksfp8\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.337023    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-socket\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337383 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.337360    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-credential-socket\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337440 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.337374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-certs\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337440 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.337402    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-data\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337544 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.337447    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-socket\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337697 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.337678    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-certs\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.337754 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.337733    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istiod-ca-cert\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.339089 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.339073    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-envoy\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.339387 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.339371    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-podinfo\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.344195 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.344173    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-token\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.344540 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.344524    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfp8\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-kube-api-access-ksfp8\") pod \"e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") " pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.428600 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.428575    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:50.704389 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.704362    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"]
Sep 11 19:50:50.705592 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:50:50.705561    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8baf6a2e_cee6_4a4c_80c3_7ff12f2337f8.slice/crio-cc3c799cfa5db27b79a02cfe8c718f37dfcc1348cca68e6cd8509a2c4f56a754 WatchSource:0}: Error finding container cc3c799cfa5db27b79a02cfe8c718f37dfcc1348cca68e6cd8509a2c4f56a754: Status 404 returned error can't find the container with id cc3c799cfa5db27b79a02cfe8c718f37dfcc1348cca68e6cd8509a2c4f56a754
Sep 11 19:50:50.708087 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.708051    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:50:50.708181 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.708161    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:50:50.708221 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:50.708208    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:50:51.427335 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:51.427299    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" event={"ID":"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8","Type":"ContainerStarted","Data":"d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e"}
Sep 11 19:50:51.427710 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:51.427391    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" event={"ID":"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8","Type":"ContainerStarted","Data":"cc3c799cfa5db27b79a02cfe8c718f37dfcc1348cca68e6cd8509a2c4f56a754"}
Sep 11 19:50:51.428766 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:51.428749    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:51.433708 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:51.433682    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:51.449008 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:51.448965    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" podStartSLOduration=1.448952075 podStartE2EDuration="1.448952075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:50:51.446720033 +0000 UTC m=+1288.122656938" watchObservedRunningTime="2026-09-11 19:50:51.448952075 +0000 UTC m=+1288.124888982"
Sep 11 19:50:52.431995 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:52.431968    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:50:52.432950 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:50:52.432929    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:51:04.054451 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:04.054423    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:51:14.551807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:14.551775    2571 generic.go:356] "Generic (PLEG): container finished" podID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerID="d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e" exitCode=0
Sep 11 19:51:14.552227 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:14.551836    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" event={"ID":"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8","Type":"ContainerDied","Data":"d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e"}
Sep 11 19:51:14.552227 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:14.552168    2571 scope.go:117] "RemoveContainer" containerID="d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e"
Sep 11 19:51:14.553292 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:14.553261    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:51:14.553355 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:14.553341    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:51:14.553406 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:14.553395    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:51:15.558759 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:15.558720    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" event={"ID":"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8","Type":"ContainerStarted","Data":"cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6"}
Sep 11 19:51:16.429216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:16.429175    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:51:16.429396 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:16.429259    2571 patch_prober.go:28] interesting pod/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 19:51:16.429396 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:16.429296    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 19:51:17.429802 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:17.429755    2571 patch_prober.go:28] interesting pod/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 19:51:17.430270 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:17.429830    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 19:51:18.429570 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:18.429528    2571 patch_prober.go:28] interesting pod/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 19:51:18.429781 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:18.429605    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 19:51:19.429281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:19.429240    2571 patch_prober.go:28] interesting pod/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 19:51:19.429664 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:19.429310    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 19:51:20.429710 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:20.429667    2571 patch_prober.go:28] interesting pod/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 19:51:20.430227 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:20.429724    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 19:51:21.433632 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:21.433600    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:51:21.434012 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:21.433842    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:51:21.434540 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:21.434516    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:51:30.468872 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.468837    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"]
Sep 11 19:51:30.469325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.469308    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="975837cd-aff1-404b-a486-d8856ce8ef11" containerName="istio-proxy"
Sep 11 19:51:30.473550 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.473526    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.476318 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.476297    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-81969518-openshift-default-dockercfg-4tz4k\""
Sep 11 19:51:30.486010 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.485986    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"]
Sep 11 19:51:30.588886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.588858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-data\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.589042 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.588890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-podinfo\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.589042 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.588914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-certs\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.589042 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.589014    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-envoy\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.589208 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.589048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a76138-64c7-4b0b-8f0d-9871bc300e56-istiod-ca-cert\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.589208 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.589145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-socket\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.589208 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.589166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-token\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.589208 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.589184    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrbd\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-kube-api-access-cdrbd\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.589346 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.589216    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-credential-socket\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.689857 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.689822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-data\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.689857 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.689857    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-podinfo\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.689881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-certs\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.689911    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-envoy\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.689932    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a76138-64c7-4b0b-8f0d-9871bc300e56-istiod-ca-cert\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.689984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-socket\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.690010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-token\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.690038    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrbd\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-kube-api-access-cdrbd\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.690073    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-credential-socket\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.690237    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-data\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690325 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.690290    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-certs\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690661 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.690643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-credential-socket\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690747 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.690642    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-socket\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.690747 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.690691    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a76138-64c7-4b0b-8f0d-9871bc300e56-istiod-ca-cert\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.692243 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.692227    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-envoy\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.692394 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.692376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-podinfo\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.698063 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.698033    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-token\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.698689 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.698670    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrbd\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-kube-api-access-cdrbd\") pod \"e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") " pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:30.783415 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:30.783359    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:31.214602 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:31.214539    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"]
Sep 11 19:51:31.217119 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:51:31.217078    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a76138_64c7_4b0b_8f0d_9871bc300e56.slice/crio-7dbf84afb46067db2c247742510afb0daffeee060f3ed416fc4203dc6ae632bb WatchSource:0}: Error finding container 7dbf84afb46067db2c247742510afb0daffeee060f3ed416fc4203dc6ae632bb: Status 404 returned error can't find the container with id 7dbf84afb46067db2c247742510afb0daffeee060f3ed416fc4203dc6ae632bb
Sep 11 19:51:31.219498 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:31.219458    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:51:31.219580 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:31.219545    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:51:31.219618 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:31.219588    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:51:31.640253 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:31.640220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" event={"ID":"87a76138-64c7-4b0b-8f0d-9871bc300e56","Type":"ContainerStarted","Data":"f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38"}
Sep 11 19:51:31.640594 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:31.640274    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" event={"ID":"87a76138-64c7-4b0b-8f0d-9871bc300e56","Type":"ContainerStarted","Data":"7dbf84afb46067db2c247742510afb0daffeee060f3ed416fc4203dc6ae632bb"}
Sep 11 19:51:31.667829 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:31.667782    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" podStartSLOduration=1.667765741 podStartE2EDuration="1.667765741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:31.666737418 +0000 UTC m=+1328.342674323" watchObservedRunningTime="2026-09-11 19:51:31.667765741 +0000 UTC m=+1328.343702647"
Sep 11 19:51:31.784368 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:31.784338    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:32.789119 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:32.789081    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:33.649893 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:33.649862    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:33.650617 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:33.650598    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:51:36.455711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:36.455677    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:51:41.657314 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.657273    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"]
Sep 11 19:51:41.660689 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.660668    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:41.665340 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.665314    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-81969518-dockercfg-fbqf9\""
Sep 11 19:51:41.665593 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.665576    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-81969518\""
Sep 11 19:51:41.679948 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.679922    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"]
Sep 11 19:51:41.787959 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.787926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6qw\" (UniqueName: \"kubernetes.io/projected/84ef5ba7-2a28-40f7-842a-8360aadaff61-kube-api-access-7v6qw\") pod \"payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc\" (UID: \"84ef5ba7-2a28-40f7-842a-8360aadaff61\") " pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:41.788202 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.787997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ef5ba7-2a28-40f7-842a-8360aadaff61-plugins-config-volume\") pod \"payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc\" (UID: \"84ef5ba7-2a28-40f7-842a-8360aadaff61\") " pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:41.888712 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.888680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ef5ba7-2a28-40f7-842a-8360aadaff61-plugins-config-volume\") pod \"payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc\" (UID: \"84ef5ba7-2a28-40f7-842a-8360aadaff61\") " pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:41.888861 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.888747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6qw\" (UniqueName: \"kubernetes.io/projected/84ef5ba7-2a28-40f7-842a-8360aadaff61-kube-api-access-7v6qw\") pod \"payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc\" (UID: \"84ef5ba7-2a28-40f7-842a-8360aadaff61\") " pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:41.889381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.889363    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ef5ba7-2a28-40f7-842a-8360aadaff61-plugins-config-volume\") pod \"payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc\" (UID: \"84ef5ba7-2a28-40f7-842a-8360aadaff61\") " pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:41.896754 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.896728    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6qw\" (UniqueName: \"kubernetes.io/projected/84ef5ba7-2a28-40f7-842a-8360aadaff61-kube-api-access-7v6qw\") pod \"payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc\" (UID: \"84ef5ba7-2a28-40f7-842a-8360aadaff61\") " pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:41.970836 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:41.970767    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:42.247151 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:42.247123    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"]
Sep 11 19:51:42.248273 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:51:42.248245    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ef5ba7_2a28_40f7_842a_8360aadaff61.slice/crio-f7075d21fa0beed11983df120612bbc732a7fc33795d9cb85e898e2fdbd39c83 WatchSource:0}: Error finding container f7075d21fa0beed11983df120612bbc732a7fc33795d9cb85e898e2fdbd39c83: Status 404 returned error can't find the container with id f7075d21fa0beed11983df120612bbc732a7fc33795d9cb85e898e2fdbd39c83
Sep 11 19:51:42.695742 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:42.695702    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc" event={"ID":"84ef5ba7-2a28-40f7-842a-8360aadaff61","Type":"ContainerStarted","Data":"9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2"}
Sep 11 19:51:42.696187 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:42.695759    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc" event={"ID":"84ef5ba7-2a28-40f7-842a-8360aadaff61","Type":"ContainerStarted","Data":"f7075d21fa0beed11983df120612bbc732a7fc33795d9cb85e898e2fdbd39c83"}
Sep 11 19:51:42.696187 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:42.695790    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:42.715205 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:42.715153    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc" podStartSLOduration=1.715134472 podStartE2EDuration="1.715134472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:42.71328704 +0000 UTC m=+1339.389223951" watchObservedRunningTime="2026-09-11 19:51:42.715134472 +0000 UTC m=+1339.391071379"
Sep 11 19:51:53.703481 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:53.703451    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:51:59.782537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:59.782509    2571 generic.go:356] "Generic (PLEG): container finished" podID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerID="f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38" exitCode=0
Sep 11 19:51:59.782889 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:59.782583    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" event={"ID":"87a76138-64c7-4b0b-8f0d-9871bc300e56","Type":"ContainerDied","Data":"f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38"}
Sep 11 19:51:59.783057 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:59.783036    2571 scope.go:117] "RemoveContainer" containerID="f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38"
Sep 11 19:51:59.784160 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:59.784130    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:51:59.784239 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:59.784197    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:51:59.784239 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:51:59.784230    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:52:00.790177 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:00.790145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" event={"ID":"87a76138-64c7-4b0b-8f0d-9871bc300e56","Type":"ContainerStarted","Data":"47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19"}
Sep 11 19:52:01.784335 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:01.784296    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:52:01.785521 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:01.785491    2571 patch_prober.go:28] interesting pod/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:52:01.785644 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:01.785544    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 19:52:02.784258 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:02.784221    2571 patch_prober.go:28] interesting pod/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:52:02.784664 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:02.784279    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 19:52:03.248280 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:03.248245    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:52:03.649920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:03.649889    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:52:03.783880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:03.783847    2571 patch_prober.go:28] interesting pod/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:52:03.784038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:03.783909    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 19:52:04.784442 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:04.784408    2571 patch_prober.go:28] interesting pod/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:52:04.784836 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:04.784462    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 19:52:05.784455 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:05.784419    2571 patch_prober.go:28] interesting pod/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:52:05.784829 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:05.784470    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 19:52:06.788784 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:06.788752    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:52:06.789831 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:06.789811    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:52:14.048007 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.047975    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"]
Sep 11 19:52:14.052220 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.052202    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.055130 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.055112    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a8ee95\"/\"e2e-conflict-a-50f6e9f6-kserve-self-signed-certs\""
Sep 11 19:52:14.061081 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.061061    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"]
Sep 11 19:52:14.160662 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.160626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tmp-dir\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.160808 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.160724    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfzr\" (UniqueName: \"kubernetes.io/projected/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-kube-api-access-ktfzr\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.160808 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.160800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tls-certs\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.160932 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.160819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-dshm\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.160932 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.160869    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-home\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.160932 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.160908    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-model-cache\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.261854 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.261824    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tls-certs\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.261854 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.261853    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-dshm\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.262066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.261875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-home\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.262066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.261897    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-model-cache\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.262066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.261932    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tmp-dir\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.262066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.262007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfzr\" (UniqueName: \"kubernetes.io/projected/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-kube-api-access-ktfzr\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.262400 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.262373    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tmp-dir\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.262491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.262429    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-model-cache\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.262491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.262433    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-home\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.264095 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.264067    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-dshm\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.264318 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.264301    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tls-certs\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.271454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.271431    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfzr\" (UniqueName: \"kubernetes.io/projected/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-kube-api-access-ktfzr\") pod \"e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.361518 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.361434    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:14.631539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.631513    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"]
Sep 11 19:52:14.632721 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:52:14.632698    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6212a4e_4f6d_4173_a6d7_18fb8c8dfee0.slice/crio-7f63781c2ee3ba5b4ce11916d06aeb11519c5c0eac01a983b24be61f67bc277f WatchSource:0}: Error finding container 7f63781c2ee3ba5b4ce11916d06aeb11519c5c0eac01a983b24be61f67bc277f: Status 404 returned error can't find the container with id 7f63781c2ee3ba5b4ce11916d06aeb11519c5c0eac01a983b24be61f67bc277f
Sep 11 19:52:14.867384 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.867347    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq" event={"ID":"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0","Type":"ContainerStarted","Data":"4832aa3ff6613bccb9f7e63fad272c3feeb2d561eaf87f57c322a4efb3dced77"}
Sep 11 19:52:14.867544 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.867394    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq" event={"ID":"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0","Type":"ContainerStarted","Data":"7f63781c2ee3ba5b4ce11916d06aeb11519c5c0eac01a983b24be61f67bc277f"}
Sep 11 19:52:14.885726 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:14.885639    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq" podStartSLOduration=0.885624938 podStartE2EDuration="885.624938ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:14.884346343 +0000 UTC m=+1371.560283251" watchObservedRunningTime="2026-09-11 19:52:14.885624938 +0000 UTC m=+1371.561561845"
Sep 11 19:52:24.361645 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:24.361617    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:24.361998 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:24.361662    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:24.373484 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:24.373461    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:24.929122 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:24.929073    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:52:25.453756 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.453723    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"]
Sep 11 19:52:25.454249 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.453975    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc" podUID="84ef5ba7-2a28-40f7-842a-8360aadaff61" containerName="payload-processing" containerID="cri-o://9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2" gracePeriod=30
Sep 11 19:52:25.706272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.706194    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:52:25.765369 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.765339    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ef5ba7-2a28-40f7-842a-8360aadaff61-plugins-config-volume\") pod \"84ef5ba7-2a28-40f7-842a-8360aadaff61\" (UID: \"84ef5ba7-2a28-40f7-842a-8360aadaff61\") "
Sep 11 19:52:25.765553 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.765390    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v6qw\" (UniqueName: \"kubernetes.io/projected/84ef5ba7-2a28-40f7-842a-8360aadaff61-kube-api-access-7v6qw\") pod \"84ef5ba7-2a28-40f7-842a-8360aadaff61\" (UID: \"84ef5ba7-2a28-40f7-842a-8360aadaff61\") "
Sep 11 19:52:25.765750 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.765727    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ef5ba7-2a28-40f7-842a-8360aadaff61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84ef5ba7-2a28-40f7-842a-8360aadaff61" (UID: "84ef5ba7-2a28-40f7-842a-8360aadaff61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:52:25.767384 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.767365    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ef5ba7-2a28-40f7-842a-8360aadaff61-kube-api-access-7v6qw" (OuterVolumeSpecName: "kube-api-access-7v6qw") pod "84ef5ba7-2a28-40f7-842a-8360aadaff61" (UID: "84ef5ba7-2a28-40f7-842a-8360aadaff61"). InnerVolumeSpecName "kube-api-access-7v6qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:52:25.866152 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.866125    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ef5ba7-2a28-40f7-842a-8360aadaff61-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:25.866152 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.866148    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6qw\" (UniqueName: \"kubernetes.io/projected/84ef5ba7-2a28-40f7-842a-8360aadaff61-kube-api-access-7v6qw\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:25.924097 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.924068    2571 generic.go:356] "Generic (PLEG): container finished" podID="84ef5ba7-2a28-40f7-842a-8360aadaff61" containerID="9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2" exitCode=0
Sep 11 19:52:25.924224 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.924135    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"
Sep 11 19:52:25.924224 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.924153    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc" event={"ID":"84ef5ba7-2a28-40f7-842a-8360aadaff61","Type":"ContainerDied","Data":"9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2"}
Sep 11 19:52:25.924224 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.924217    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc" event={"ID":"84ef5ba7-2a28-40f7-842a-8360aadaff61","Type":"ContainerDied","Data":"f7075d21fa0beed11983df120612bbc732a7fc33795d9cb85e898e2fdbd39c83"}
Sep 11 19:52:25.924385 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.924260    2571 scope.go:117] "RemoveContainer" containerID="9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2"
Sep 11 19:52:25.933306 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.933289    2571 scope.go:117] "RemoveContainer" containerID="9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2"
Sep 11 19:52:25.933574 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:52:25.933556    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2\": container with ID starting with 9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2 not found: ID does not exist" containerID="9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2"
Sep 11 19:52:25.933655 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.933579    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2"} err="failed to get container status \"9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2\": rpc error: code = NotFound desc = could not find container \"9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2\": container with ID starting with 9ca1a52b0c66a6b182e433824027d75574062043114197c1e7fd967f77ddb7d2 not found: ID does not exist"
Sep 11 19:52:25.947637 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.947613    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"]
Sep 11 19:52:25.950075 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:25.950056    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81969518-6c6bfd774f-j6prc"]
Sep 11 19:52:27.873281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:27.873250    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84ef5ba7-2a28-40f7-842a-8360aadaff61" path="/var/lib/kubelet/pods/84ef5ba7-2a28-40f7-842a-8360aadaff61/volumes"
Sep 11 19:52:32.394642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:32.394607    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"]
Sep 11 19:52:32.395026 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:32.394807    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy" containerID="cri-o://47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19" gracePeriod=30
Sep 11 19:52:34.854697 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:34.854664    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:52:36.791874 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:36.791835    2571 patch_prober.go:28] interesting pod/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 19:52:36.792395 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:36.791915    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:52:37.641309 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.641281    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:52:37.773539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773446    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-data\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773492    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-token\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773522    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdrbd\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-kube-api-access-cdrbd\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773581    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-socket\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773628    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-certs\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773657    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-podinfo\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773683    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a76138-64c7-4b0b-8f0d-9871bc300e56-istiod-ca-cert\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773702    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-credential-socket\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773731    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-envoy\") pod \"87a76138-64c7-4b0b-8f0d-9871bc300e56\" (UID: \"87a76138-64c7-4b0b-8f0d-9871bc300e56\") "
Sep 11 19:52:37.773840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773759    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-data" (OuterVolumeSpecName: "istio-data") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:52:37.774237 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.773898    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:52:37.774237 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.774155    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:52:37.774237 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.774165    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.774237 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.774204    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.774237 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.774220    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a76138-64c7-4b0b-8f0d-9871bc300e56-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:52:37.774482 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.774353    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:52:37.775981 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.775956    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-kube-api-access-cdrbd" (OuterVolumeSpecName: "kube-api-access-cdrbd") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "kube-api-access-cdrbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:52:37.776079 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.775992    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:52:37.776079 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.776003    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-token" (OuterVolumeSpecName: "istio-token") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:52:37.776079 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.775998    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87a76138-64c7-4b0b-8f0d-9871bc300e56" (UID: "87a76138-64c7-4b0b-8f0d-9871bc300e56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:52:37.875504 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.875466    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.875504 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.875501    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.875920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.875515    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a76138-64c7-4b0b-8f0d-9871bc300e56-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.875920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.875529    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.875920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.875543    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.875920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.875557    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.875920 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.875571    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrbd\" (UniqueName: \"kubernetes.io/projected/87a76138-64c7-4b0b-8f0d-9871bc300e56-kube-api-access-cdrbd\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:52:37.988426 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.988387    2571 generic.go:356] "Generic (PLEG): container finished" podID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerID="47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19" exitCode=0
Sep 11 19:52:37.988585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.988435    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" event={"ID":"87a76138-64c7-4b0b-8f0d-9871bc300e56","Type":"ContainerDied","Data":"47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19"}
Sep 11 19:52:37.988585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.988443    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"
Sep 11 19:52:37.988585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.988480    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz" event={"ID":"87a76138-64c7-4b0b-8f0d-9871bc300e56","Type":"ContainerDied","Data":"7dbf84afb46067db2c247742510afb0daffeee060f3ed416fc4203dc6ae632bb"}
Sep 11 19:52:37.988585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.988503    2571 scope.go:117] "RemoveContainer" containerID="47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19"
Sep 11 19:52:37.997714 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:37.997692    2571 scope.go:117] "RemoveContainer" containerID="f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38"
Sep 11 19:52:38.006377 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:38.006353    2571 scope.go:117] "RemoveContainer" containerID="47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19"
Sep 11 19:52:38.006694 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:52:38.006675    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19\": container with ID starting with 47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19 not found: ID does not exist" containerID="47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19"
Sep 11 19:52:38.006770 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:38.006708    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19"} err="failed to get container status \"47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19\": rpc error: code = NotFound desc = could not find container \"47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19\": container with ID starting with 47692c6a3b2f2834294b5eda8aed20e9bbc4907f1291685fb38b07e3030bfa19 not found: ID does not exist"
Sep 11 19:52:38.006770 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:38.006731    2571 scope.go:117] "RemoveContainer" containerID="f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38"
Sep 11 19:52:38.006868 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:38.006812    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"]
Sep 11 19:52:38.007033 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:52:38.007016    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38\": container with ID starting with f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38 not found: ID does not exist" containerID="f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38"
Sep 11 19:52:38.007073 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:38.007039    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38"} err="failed to get container status \"f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38\": rpc error: code = NotFound desc = could not find container \"f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38\": container with ID starting with f24017cba276bf4c6520688e3154be2d1e923c6a5bc2efc4ec2662752055ea38 not found: ID does not exist"
Sep 11 19:52:38.010306 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:38.010285    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81969518-openshift-default-78bcbcc5bd-d6mgz"]
Sep 11 19:52:39.873537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:39.873501    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" path="/var/lib/kubelet/pods/87a76138-64c7-4b0b-8f0d-9871bc300e56/volumes"
Sep 11 19:52:44.863362 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.863328    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"]
Sep 11 19:52:44.863731 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.863664    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy"
Sep 11 19:52:44.863731 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.863677    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy"
Sep 11 19:52:44.863731 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.863689    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ef5ba7-2a28-40f7-842a-8360aadaff61" containerName="payload-processing"
Sep 11 19:52:44.863731 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.863695    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ef5ba7-2a28-40f7-842a-8360aadaff61" containerName="payload-processing"
Sep 11 19:52:44.863861 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.863769    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ef5ba7-2a28-40f7-842a-8360aadaff61" containerName="payload-processing"
Sep 11 19:52:44.863861 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.863779    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy"
Sep 11 19:52:44.863861 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.863790    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy"
Sep 11 19:52:44.868359 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.868341    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:44.872230 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.872208    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f76076\"/\"kube-root-ca.crt\""
Sep 11 19:52:44.872351 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.872253    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f76076\"/\"openshift-service-ca.crt\""
Sep 11 19:52:44.872351 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.872250    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f76076\"/\"explicit-model-f76076-kserve-self-signed-certs\""
Sep 11 19:52:44.872461 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.872348    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f76076\"/\"default-dockercfg-5j7f5\""
Sep 11 19:52:44.878937 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.878915    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"]
Sep 11 19:52:44.942483 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.942450    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-tmp-dir\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:44.942672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.942487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkgzs\" (UniqueName: \"kubernetes.io/projected/8708e450-9f66-46f0-aa40-b94470952a2e-kube-api-access-zkgzs\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:44.942672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.942526    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8708e450-9f66-46f0-aa40-b94470952a2e-tls-certs\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:44.942672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.942561    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-dshm\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:44.942672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.942650    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-home\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:44.942808 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:44.942683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-model-cache\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.043471 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043430    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-home\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.043471 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043475    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-model-cache\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.043698 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-tmp-dir\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.043698 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043543    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkgzs\" (UniqueName: \"kubernetes.io/projected/8708e450-9f66-46f0-aa40-b94470952a2e-kube-api-access-zkgzs\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.043698 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8708e450-9f66-46f0-aa40-b94470952a2e-tls-certs\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.043698 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043611    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-dshm\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.043912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043886    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-home\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.043956 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043928    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-model-cache\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.044006 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.043966    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-tmp-dir\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.045847 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.045824    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-dshm\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.046014 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.045994    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8708e450-9f66-46f0-aa40-b94470952a2e-tls-certs\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.052399 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.052380    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkgzs\" (UniqueName: \"kubernetes.io/projected/8708e450-9f66-46f0-aa40-b94470952a2e-kube-api-access-zkgzs\") pod \"explicit-model-f76076-kserve-bc449f569-2rmj2\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") " pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.177699 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.177656    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:45.454094 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:45.454062    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"]
Sep 11 19:52:45.455455 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:52:45.455420    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8708e450_9f66_46f0_aa40_b94470952a2e.slice/crio-44280caef918f8d66f88c5ee7b369558d196b80946ce4f4861423bc0ca43890a WatchSource:0}: Error finding container 44280caef918f8d66f88c5ee7b369558d196b80946ce4f4861423bc0ca43890a: Status 404 returned error can't find the container with id 44280caef918f8d66f88c5ee7b369558d196b80946ce4f4861423bc0ca43890a
Sep 11 19:52:46.034455 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:46.034415    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2" event={"ID":"8708e450-9f66-46f0-aa40-b94470952a2e","Type":"ContainerStarted","Data":"ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d"}
Sep 11 19:52:46.034932 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:46.034464    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2" event={"ID":"8708e450-9f66-46f0-aa40-b94470952a2e","Type":"ContainerStarted","Data":"44280caef918f8d66f88c5ee7b369558d196b80946ce4f4861423bc0ca43890a"}
Sep 11 19:52:53.192660 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.192602    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2" podStartSLOduration=9.192582789 podStartE2EDuration="9.192582789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:46.051933578 +0000 UTC m=+1402.727870490" watchObservedRunningTime="2026-09-11 19:52:53.192582789 +0000 UTC m=+1409.868519746"
Sep 11 19:52:53.194278 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.194258    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"]
Sep 11 19:52:53.194599 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.194585    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy"
Sep 11 19:52:53.194639 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.194601    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a76138-64c7-4b0b-8f0d-9871bc300e56" containerName="istio-proxy"
Sep 11 19:52:53.197737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.197721    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:53.202533 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.202514    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e8cfcf\""
Sep 11 19:52:53.203524 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.203504    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e8cfcf-dockercfg-4b49b\""
Sep 11 19:52:53.210510 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.210479    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv\" (UID: \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:53.210611 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.210589    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7bc\" (UniqueName: \"kubernetes.io/projected/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-kube-api-access-2k7bc\") pod \"payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv\" (UID: \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:53.226893 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.226871    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"]
Sep 11 19:52:53.312114 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.312075    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7bc\" (UniqueName: \"kubernetes.io/projected/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-kube-api-access-2k7bc\") pod \"payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv\" (UID: \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:53.312328 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.312207    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv\" (UID: \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:53.312881 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.312861    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv\" (UID: \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:53.320917 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.320884    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7bc\" (UniqueName: \"kubernetes.io/projected/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-kube-api-access-2k7bc\") pod \"payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv\" (UID: \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:53.508098 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.507996    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:53.778904 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:53.778875    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"]
Sep 11 19:52:53.780060 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:52:53.780027    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd29eeb3_6957_4a8e_80f8_14d42ce6eeac.slice/crio-36dedb18d84937f07c699a56adf21e6fad8ee3e66c6a8f226fd0c241f905a247 WatchSource:0}: Error finding container 36dedb18d84937f07c699a56adf21e6fad8ee3e66c6a8f226fd0c241f905a247: Status 404 returned error can't find the container with id 36dedb18d84937f07c699a56adf21e6fad8ee3e66c6a8f226fd0c241f905a247
Sep 11 19:52:54.086688 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:54.086597    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv" event={"ID":"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac","Type":"ContainerStarted","Data":"2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79"}
Sep 11 19:52:54.086688 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:54.086651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv" event={"ID":"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac","Type":"ContainerStarted","Data":"36dedb18d84937f07c699a56adf21e6fad8ee3e66c6a8f226fd0c241f905a247"}
Sep 11 19:52:54.086891 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:54.086730    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:52:54.109316 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:54.109265    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv" podStartSLOduration=1.109249303 podStartE2EDuration="1.109249303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:54.107441902 +0000 UTC m=+1410.783378813" watchObservedRunningTime="2026-09-11 19:52:54.109249303 +0000 UTC m=+1410.785186251"
Sep 11 19:52:55.178228 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:55.178190    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:55.178686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:55.178460    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:55.191481 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:55.191450    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:52:55.956296 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:55.956260    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:52:56.108122 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:52:56.108070    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:53:05.093866 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:05.093828    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 19:53:21.628922 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:21.628886    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"]
Sep 11 19:53:23.131245 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:23.131206    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"]
Sep 11 19:53:27.765067 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:27.765028    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"]
Sep 11 19:53:27.769826 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:27.769807    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:27.772610 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:27.772580    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f7fd24\""
Sep 11 19:53:27.772726 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:27.772618    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f7fd24-dockercfg-xzz2h\""
Sep 11 19:53:27.783037 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:27.783009    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"]
Sep 11 19:53:27.948406 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:27.948372    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f705fabd-4ccc-4926-af16-43cdef7d4f27-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp\" (UID: \"f705fabd-4ccc-4926-af16-43cdef7d4f27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:27.948583 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:27.948442    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsh4\" (UniqueName: \"kubernetes.io/projected/f705fabd-4ccc-4926-af16-43cdef7d4f27-kube-api-access-9bsh4\") pod \"payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp\" (UID: \"f705fabd-4ccc-4926-af16-43cdef7d4f27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:28.049167 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:28.049053    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f705fabd-4ccc-4926-af16-43cdef7d4f27-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp\" (UID: \"f705fabd-4ccc-4926-af16-43cdef7d4f27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:28.049167 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:28.049142    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsh4\" (UniqueName: \"kubernetes.io/projected/f705fabd-4ccc-4926-af16-43cdef7d4f27-kube-api-access-9bsh4\") pod \"payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp\" (UID: \"f705fabd-4ccc-4926-af16-43cdef7d4f27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:28.049902 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:28.049881    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f705fabd-4ccc-4926-af16-43cdef7d4f27-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp\" (UID: \"f705fabd-4ccc-4926-af16-43cdef7d4f27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:28.058966 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:28.058936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsh4\" (UniqueName: \"kubernetes.io/projected/f705fabd-4ccc-4926-af16-43cdef7d4f27-kube-api-access-9bsh4\") pod \"payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp\" (UID: \"f705fabd-4ccc-4926-af16-43cdef7d4f27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:28.079814 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:28.079786    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:28.353977 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:28.353930    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"]
Sep 11 19:53:28.355182 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:53:28.355146    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf705fabd_4ccc_4926_af16_43cdef7d4f27.slice/crio-b3213c7f17ae13975f57542fddedc0e578222d224c8412b1460686cbf440adbd WatchSource:0}: Error finding container b3213c7f17ae13975f57542fddedc0e578222d224c8412b1460686cbf440adbd: Status 404 returned error can't find the container with id b3213c7f17ae13975f57542fddedc0e578222d224c8412b1460686cbf440adbd
Sep 11 19:53:29.249471 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:29.249420    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:53:29.275129 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:29.275081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp" event={"ID":"f705fabd-4ccc-4926-af16-43cdef7d4f27","Type":"ContainerStarted","Data":"f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b"}
Sep 11 19:53:29.275323 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:29.275152    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp" event={"ID":"f705fabd-4ccc-4926-af16-43cdef7d4f27","Type":"ContainerStarted","Data":"b3213c7f17ae13975f57542fddedc0e578222d224c8412b1460686cbf440adbd"}
Sep 11 19:53:29.275323 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:29.275268    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:29.297173 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:29.297125    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp" podStartSLOduration=2.297110259 podStartE2EDuration="2.297110259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:29.294139464 +0000 UTC m=+1445.970076363" watchObservedRunningTime="2026-09-11 19:53:29.297110259 +0000 UTC m=+1445.973047159"
Sep 11 19:53:36.647932 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.647882    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2" podUID="8708e450-9f66-46f0-aa40-b94470952a2e" containerName="main" containerID="cri-o://ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d" gracePeriod=45
Sep 11 19:53:36.895229 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.895199    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:53:36.927358 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927274    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-home\") pod \"8708e450-9f66-46f0-aa40-b94470952a2e\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") "
Sep 11 19:53:36.927519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927375    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-model-cache\") pod \"8708e450-9f66-46f0-aa40-b94470952a2e\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") "
Sep 11 19:53:36.927519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927428    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8708e450-9f66-46f0-aa40-b94470952a2e-tls-certs\") pod \"8708e450-9f66-46f0-aa40-b94470952a2e\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") "
Sep 11 19:53:36.927519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927463    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-tmp-dir\") pod \"8708e450-9f66-46f0-aa40-b94470952a2e\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") "
Sep 11 19:53:36.927519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927511    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-dshm\") pod \"8708e450-9f66-46f0-aa40-b94470952a2e\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") "
Sep 11 19:53:36.927735 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927567    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkgzs\" (UniqueName: \"kubernetes.io/projected/8708e450-9f66-46f0-aa40-b94470952a2e-kube-api-access-zkgzs\") pod \"8708e450-9f66-46f0-aa40-b94470952a2e\" (UID: \"8708e450-9f66-46f0-aa40-b94470952a2e\") "
Sep 11 19:53:36.927983 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927562    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-home" (OuterVolumeSpecName: "home") pod "8708e450-9f66-46f0-aa40-b94470952a2e" (UID: "8708e450-9f66-46f0-aa40-b94470952a2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:36.927983 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927930    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:36.927983 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927595    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "8708e450-9f66-46f0-aa40-b94470952a2e" (UID: "8708e450-9f66-46f0-aa40-b94470952a2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:36.927983 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.927737    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8708e450-9f66-46f0-aa40-b94470952a2e" (UID: "8708e450-9f66-46f0-aa40-b94470952a2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:36.929813 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.929787    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8708e450-9f66-46f0-aa40-b94470952a2e-kube-api-access-zkgzs" (OuterVolumeSpecName: "kube-api-access-zkgzs") pod "8708e450-9f66-46f0-aa40-b94470952a2e" (UID: "8708e450-9f66-46f0-aa40-b94470952a2e"). InnerVolumeSpecName "kube-api-access-zkgzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:53:36.930052 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.930025    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-dshm" (OuterVolumeSpecName: "dshm") pod "8708e450-9f66-46f0-aa40-b94470952a2e" (UID: "8708e450-9f66-46f0-aa40-b94470952a2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:36.930116 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:36.930017    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8708e450-9f66-46f0-aa40-b94470952a2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8708e450-9f66-46f0-aa40-b94470952a2e" (UID: "8708e450-9f66-46f0-aa40-b94470952a2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:53:37.028449 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.028408    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8708e450-9f66-46f0-aa40-b94470952a2e-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:37.028449 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.028443    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:37.028449 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.028454    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:37.028694 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.028466    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkgzs\" (UniqueName: \"kubernetes.io/projected/8708e450-9f66-46f0-aa40-b94470952a2e-kube-api-access-zkgzs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:37.028694 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.028479    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8708e450-9f66-46f0-aa40-b94470952a2e-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:37.318532 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.318436    2571 generic.go:356] "Generic (PLEG): container finished" podID="8708e450-9f66-46f0-aa40-b94470952a2e" containerID="ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d" exitCode=0
Sep 11 19:53:37.318532 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.318484    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"
Sep 11 19:53:37.318759 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.318530    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2" event={"ID":"8708e450-9f66-46f0-aa40-b94470952a2e","Type":"ContainerDied","Data":"ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d"}
Sep 11 19:53:37.318759 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.318574    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2" event={"ID":"8708e450-9f66-46f0-aa40-b94470952a2e","Type":"ContainerDied","Data":"44280caef918f8d66f88c5ee7b369558d196b80946ce4f4861423bc0ca43890a"}
Sep 11 19:53:37.318759 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.318593    2571 scope.go:117] "RemoveContainer" containerID="ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d"
Sep 11 19:53:37.328345 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.328326    2571 scope.go:117] "RemoveContainer" containerID="ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d"
Sep 11 19:53:37.328613 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:37.328595    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d\": container with ID starting with ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d not found: ID does not exist" containerID="ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d"
Sep 11 19:53:37.328661 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.328623    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d"} err="failed to get container status \"ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d\": rpc error: code = NotFound desc = could not find container \"ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d\": container with ID starting with ed02a72d275e37ace9f4de697991ebed027ce4b3210720f4eef95c56fd61041d not found: ID does not exist"
Sep 11 19:53:37.341584 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.341556    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"]
Sep 11 19:53:37.345343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.345321    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f76076/explicit-model-f76076-kserve-bc449f569-2rmj2"]
Sep 11 19:53:37.874439 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:37.874406    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8708e450-9f66-46f0-aa40-b94470952a2e" path="/var/lib/kubelet/pods/8708e450-9f66-46f0-aa40-b94470952a2e/volumes"
Sep 11 19:53:38.151174 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.151054    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh" podUID="6db85eb3-e8bb-42d3-b923-cdbe18111b57" containerName="main" containerID="cri-o://eae95f350d3029ffea9f12ba5293d485e5218334d09b1fde537200947f80ffb6" gracePeriod=45
Sep 11 19:53:38.327805 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.327767    2571 generic.go:356] "Generic (PLEG): container finished" podID="6db85eb3-e8bb-42d3-b923-cdbe18111b57" containerID="eae95f350d3029ffea9f12ba5293d485e5218334d09b1fde537200947f80ffb6" exitCode=0
Sep 11 19:53:38.327960 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.327851    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh" event={"ID":"6db85eb3-e8bb-42d3-b923-cdbe18111b57","Type":"ContainerDied","Data":"eae95f350d3029ffea9f12ba5293d485e5218334d09b1fde537200947f80ffb6"}
Sep 11 19:53:38.392925 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.392900    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:53:38.440491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.440409    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tls-certs\") pod \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") "
Sep 11 19:53:38.440491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.440442    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-dshm\") pod \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") "
Sep 11 19:53:38.440704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.440495    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm9ls\" (UniqueName: \"kubernetes.io/projected/6db85eb3-e8bb-42d3-b923-cdbe18111b57-kube-api-access-xm9ls\") pod \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") "
Sep 11 19:53:38.440704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.440624    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-home\") pod \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") "
Sep 11 19:53:38.440704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.440669    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-model-cache\") pod \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") "
Sep 11 19:53:38.440876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.440767    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tmp-dir\") pod \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\" (UID: \"6db85eb3-e8bb-42d3-b923-cdbe18111b57\") "
Sep 11 19:53:38.440932 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.440902    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-home" (OuterVolumeSpecName: "home") pod "6db85eb3-e8bb-42d3-b923-cdbe18111b57" (UID: "6db85eb3-e8bb-42d3-b923-cdbe18111b57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:38.441007 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.440962    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-model-cache" (OuterVolumeSpecName: "model-cache") pod "6db85eb3-e8bb-42d3-b923-cdbe18111b57" (UID: "6db85eb3-e8bb-42d3-b923-cdbe18111b57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:38.441066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.441032    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6db85eb3-e8bb-42d3-b923-cdbe18111b57" (UID: "6db85eb3-e8bb-42d3-b923-cdbe18111b57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:38.441170 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.441152    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:38.441225 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.441175    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:38.441225 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.441191    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:38.442717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.442686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-dshm" (OuterVolumeSpecName: "dshm") pod "6db85eb3-e8bb-42d3-b923-cdbe18111b57" (UID: "6db85eb3-e8bb-42d3-b923-cdbe18111b57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:38.442717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.442703    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db85eb3-e8bb-42d3-b923-cdbe18111b57-kube-api-access-xm9ls" (OuterVolumeSpecName: "kube-api-access-xm9ls") pod "6db85eb3-e8bb-42d3-b923-cdbe18111b57" (UID: "6db85eb3-e8bb-42d3-b923-cdbe18111b57"). InnerVolumeSpecName "kube-api-access-xm9ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:53:38.442856 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.442713    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6db85eb3-e8bb-42d3-b923-cdbe18111b57" (UID: "6db85eb3-e8bb-42d3-b923-cdbe18111b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:53:38.541943 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.541905    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db85eb3-e8bb-42d3-b923-cdbe18111b57-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:38.541943 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.541936    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db85eb3-e8bb-42d3-b923-cdbe18111b57-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:38.541943 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.541946    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm9ls\" (UniqueName: \"kubernetes.io/projected/6db85eb3-e8bb-42d3-b923-cdbe18111b57-kube-api-access-xm9ls\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:38.556946 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.556913    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7497-53c6fc"]
Sep 11 19:53:38.557296 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.557282    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8708e450-9f66-46f0-aa40-b94470952a2e" containerName="main"
Sep 11 19:53:38.557343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.557298    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8708e450-9f66-46f0-aa40-b94470952a2e" containerName="main"
Sep 11 19:53:38.557343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.557317    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db85eb3-e8bb-42d3-b923-cdbe18111b57" containerName="main"
Sep 11 19:53:38.557343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.557322    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db85eb3-e8bb-42d3-b923-cdbe18111b57" containerName="main"
Sep 11 19:53:38.557441 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.557372    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db85eb3-e8bb-42d3-b923-cdbe18111b57" containerName="main"
Sep 11 19:53:38.557441 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.557382    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8708e450-9f66-46f0-aa40-b94470952a2e" containerName="main"
Sep 11 19:53:38.560423 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.560405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:38.563455 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.563425    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-76rlb\""
Sep 11 19:53:38.565777 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.565756    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7497-53c6fc"]
Sep 11 19:53:38.642503 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.642464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbts\" (UniqueName: \"kubernetes.io/projected/a2dc72d3-39ea-4160-8e2c-7efeae1ccbde-kube-api-access-4sbts\") pod \"test-curl-7497-53c6fc\" (UID: \"a2dc72d3-39ea-4160-8e2c-7efeae1ccbde\") " pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:38.743151 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.743049    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbts\" (UniqueName: \"kubernetes.io/projected/a2dc72d3-39ea-4160-8e2c-7efeae1ccbde-kube-api-access-4sbts\") pod \"test-curl-7497-53c6fc\" (UID: \"a2dc72d3-39ea-4160-8e2c-7efeae1ccbde\") " pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:38.751401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.751380    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbts\" (UniqueName: \"kubernetes.io/projected/a2dc72d3-39ea-4160-8e2c-7efeae1ccbde-kube-api-access-4sbts\") pod \"test-curl-7497-53c6fc\" (UID: \"a2dc72d3-39ea-4160-8e2c-7efeae1ccbde\") " pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:38.870979 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:38.870953    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:39.144726 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.142724    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7497-53c6fc"]
Sep 11 19:53:39.148318 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:53:39.148287    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2dc72d3_39ea_4160_8e2c_7efeae1ccbde.slice/crio-5430cc94bc460bbcb06128ac2bc45ef07defa9bfd059952c524026c186f5c054 WatchSource:0}: Error finding container 5430cc94bc460bbcb06128ac2bc45ef07defa9bfd059952c524026c186f5c054: Status 404 returned error can't find the container with id 5430cc94bc460bbcb06128ac2bc45ef07defa9bfd059952c524026c186f5c054
Sep 11 19:53:39.335399 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.335358    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7497-53c6fc" event={"ID":"a2dc72d3-39ea-4160-8e2c-7efeae1ccbde","Type":"ContainerStarted","Data":"baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52"}
Sep 11 19:53:39.335399 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.335402    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7497-53c6fc" event={"ID":"a2dc72d3-39ea-4160-8e2c-7efeae1ccbde","Type":"ContainerStarted","Data":"5430cc94bc460bbcb06128ac2bc45ef07defa9bfd059952c524026c186f5c054"}
Sep 11 19:53:39.336661 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.336644    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"
Sep 11 19:53:39.336751 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.336665    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh" event={"ID":"6db85eb3-e8bb-42d3-b923-cdbe18111b57","Type":"ContainerDied","Data":"c3e7031f4c2df0a9f7d81521aa804a35c7a27e7794bcddbedfe2770a51143c60"}
Sep 11 19:53:39.336751 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.336710    2571 scope.go:117] "RemoveContainer" containerID="eae95f350d3029ffea9f12ba5293d485e5218334d09b1fde537200947f80ffb6"
Sep 11 19:53:39.351696 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.351655    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7497-53c6fc" podStartSLOduration=1.351639545 podStartE2EDuration="1.351639545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:39.350198805 +0000 UTC m=+1456.026135713" watchObservedRunningTime="2026-09-11 19:53:39.351639545 +0000 UTC m=+1456.027576452"
Sep 11 19:53:39.363874 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.363844    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"]
Sep 11 19:53:39.367772 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.367749    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a58a49/test-model-a58a49-kserve-8d5c55f49-5q8nh"]
Sep 11 19:53:39.873554 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.873525    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db85eb3-e8bb-42d3-b923-cdbe18111b57" path="/var/lib/kubelet/pods/6db85eb3-e8bb-42d3-b923-cdbe18111b57/volumes"
Sep 11 19:53:39.914222 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.914195    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7497-53c6fc"]
Sep 11 19:53:39.917917 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:39.917893    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7497-53c6fc"]
Sep 11 19:53:40.029116 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.029081    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7497-b324af"]
Sep 11 19:53:40.029436 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.029422    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" containerName="test-curl-7497-53c6fc"
Sep 11 19:53:40.029498 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.029438    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" containerName="test-curl-7497-53c6fc"
Sep 11 19:53:40.029498 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.029493    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" containerName="test-curl-7497-53c6fc"
Sep 11 19:53:40.033925 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.033897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:40.036775 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.036750    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7497-b324af"]
Sep 11 19:53:40.056457 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.056433    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzt47\" (UniqueName: \"kubernetes.io/projected/a9076853-d7e1-4260-ba51-0018d07b3928-kube-api-access-mzt47\") pod \"test-curl-7497-b324af\" (UID: \"a9076853-d7e1-4260-ba51-0018d07b3928\") " pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:40.157881 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.157818    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzt47\" (UniqueName: \"kubernetes.io/projected/a9076853-d7e1-4260-ba51-0018d07b3928-kube-api-access-mzt47\") pod \"test-curl-7497-b324af\" (UID: \"a9076853-d7e1-4260-ba51-0018d07b3928\") " pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:40.165852 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.165835    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzt47\" (UniqueName: \"kubernetes.io/projected/a9076853-d7e1-4260-ba51-0018d07b3928-kube-api-access-mzt47\") pod \"test-curl-7497-b324af\" (UID: \"a9076853-d7e1-4260-ba51-0018d07b3928\") " pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:40.282823 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.282799    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 19:53:40.343489 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.343460    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:40.343656 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:40.343641    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7497-53c6fc" container="test-curl-7497-53c6fc"
Sep 11 19:53:40.610963 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:40.610940    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7497-b324af"]
Sep 11 19:53:40.612975 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:53:40.612940    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9076853_d7e1_4260_ba51_0018d07b3928.slice/crio-7f2640c5b5eb7823ebd0145c8b9539a9fafc6862c12e57ea9d18f35a5100ec7a WatchSource:0}: Error finding container 7f2640c5b5eb7823ebd0145c8b9539a9fafc6862c12e57ea9d18f35a5100ec7a: Status 404 returned error can't find the container with id 7f2640c5b5eb7823ebd0145c8b9539a9fafc6862c12e57ea9d18f35a5100ec7a
Sep 11 19:53:41.350560 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:41.350506    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7497-b324af" event={"ID":"a9076853-d7e1-4260-ba51-0018d07b3928","Type":"ContainerStarted","Data":"1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2"}
Sep 11 19:53:41.351000 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:41.350569    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7497-b324af" event={"ID":"a9076853-d7e1-4260-ba51-0018d07b3928","Type":"ContainerStarted","Data":"7f2640c5b5eb7823ebd0145c8b9539a9fafc6862c12e57ea9d18f35a5100ec7a"}
Sep 11 19:53:41.351000 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:41.350608    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7497-53c6fc" podUID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" containerName="test-curl-7497-53c6fc" containerID="cri-o://baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52" gracePeriod=2
Sep 11 19:53:41.353009 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:41.352982    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7497-53c6fc\" is forbidden: User \"system:node:ip-10-0-136-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-27.ec2.internal' and this object" podUID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:41.369130 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:41.369082    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7497-53c6fc\" is forbidden: User \"system:node:ip-10-0-136-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-27.ec2.internal' and this object" podUID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:41.369260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:41.369222    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7497-b324af" podStartSLOduration=1.369207308 podStartE2EDuration="1.369207308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:41.367019483 +0000 UTC m=+1458.042956382" watchObservedRunningTime="2026-09-11 19:53:41.369207308 +0000 UTC m=+1458.045144221"
Sep 11 19:53:41.759371 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:41.759339    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"]
Sep 11 19:53:41.759576 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:41.759553    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy" containerID="cri-o://cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6" gracePeriod=30
Sep 11 19:53:41.795680 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:41.795646    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7497-53c6fc\" is forbidden: User \"system:node:ip-10-0-136-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-27.ec2.internal' and this object" podUID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:41.952999 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:41.952966    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7497-b324af"]
Sep 11 19:53:41.957481 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:41.957455    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7497-b324af"]
Sep 11 19:53:42.232928 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.232898    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"]
Sep 11 19:53:42.357143 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:42.357096    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7497-b324af" container="test-curl-7497-b324af"
Sep 11 19:53:42.452089 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.452051    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"]
Sep 11 19:53:42.452477 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.452458    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9076853-d7e1-4260-ba51-0018d07b3928" containerName="test-curl-7497-b324af"
Sep 11 19:53:42.452568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.452481    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9076853-d7e1-4260-ba51-0018d07b3928" containerName="test-curl-7497-b324af"
Sep 11 19:53:42.452568 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.452564    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9076853-d7e1-4260-ba51-0018d07b3928" containerName="test-curl-7497-b324af"
Sep 11 19:53:42.455718 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.455694    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.458501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.458481    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d46ae3-openshift-default-dockercfg-7wkn5\""
Sep 11 19:53:42.467319 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.467294    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"]
Sep 11 19:53:42.580053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.579981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66dd389d-11d6-4b26-abc9-688860246c31-istio-podinfo\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.580053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.580016    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-envoy\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.580053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.580032    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-data\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.580053 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.580050    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-credential-socket\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.580321 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.580166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-socket\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.580321 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.580204    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcstt\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-kube-api-access-jcstt\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.580321 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.580256    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-certs\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.580321 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.580277    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66dd389d-11d6-4b26-abc9-688860246c31-istiod-ca-cert\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.580502 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.580371    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-istio-token\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681439 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681409    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-istio-token\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681562 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66dd389d-11d6-4b26-abc9-688860246c31-istio-podinfo\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681714 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681697    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-envoy\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681757 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681725    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-data\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681757 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-credential-socket\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681777    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-socket\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcstt\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-kube-api-access-jcstt\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681965 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681864    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-certs\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.681965 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.681894    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66dd389d-11d6-4b26-abc9-688860246c31-istiod-ca-cert\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.682338 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.682179    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-credential-socket\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.682338 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.682219    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-data\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.682338 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.682284    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-socket\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.682491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.682362    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-certs\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.682567 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.682547    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66dd389d-11d6-4b26-abc9-688860246c31-istiod-ca-cert\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.683854 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.683831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-envoy\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.683960 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.683942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66dd389d-11d6-4b26-abc9-688860246c31-istio-podinfo\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.689833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.689810    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcstt\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-kube-api-access-jcstt\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.689833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.689827    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-istio-token\") pod \"e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") " pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:42.766216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:42.766190    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:43.041816 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.041790    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"]
Sep 11 19:53:43.043388 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:53:43.043343    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dd389d_11d6_4b26_abc9_688860246c31.slice/crio-9bf1a536531a8bd4b07df0f32f6600bc9b62c9634a1e5d4df46b504c569baf78 WatchSource:0}: Error finding container 9bf1a536531a8bd4b07df0f32f6600bc9b62c9634a1e5d4df46b504c569baf78: Status 404 returned error can't find the container with id 9bf1a536531a8bd4b07df0f32f6600bc9b62c9634a1e5d4df46b504c569baf78
Sep 11 19:53:43.045767 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.045732    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:53:43.045879 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.045808    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:53:43.045879 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.045851    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:53:43.366299 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.366271    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5" event={"ID":"66dd389d-11d6-4b26-abc9-688860246c31","Type":"ContainerStarted","Data":"110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129"}
Sep 11 19:53:43.366602 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.366324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5" event={"ID":"66dd389d-11d6-4b26-abc9-688860246c31","Type":"ContainerStarted","Data":"9bf1a536531a8bd4b07df0f32f6600bc9b62c9634a1e5d4df46b504c569baf78"}
Sep 11 19:53:43.366602 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.366422    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7497-b324af" podUID="a9076853-d7e1-4260-ba51-0018d07b3928" containerName="test-curl-7497-b324af" containerID="cri-o://1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2" gracePeriod=2
Sep 11 19:53:43.389836 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:43.389800    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7497-b324af\" is forbidden: User \"system:node:ip-10-0-136-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-27.ec2.internal' and this object" podUID="a9076853-d7e1-4260-ba51-0018d07b3928" pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:43.390043 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.390006    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5" podStartSLOduration=1.389991982 podStartE2EDuration="1.389991982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:43.38758545 +0000 UTC m=+1460.063522361" watchObservedRunningTime="2026-09-11 19:53:43.389991982 +0000 UTC m=+1460.065928890"
Sep 11 19:53:43.499473 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.499452    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:43.501687 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:43.501661    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7497-b324af\" is forbidden: User \"system:node:ip-10-0-136-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-27.ec2.internal' and this object" podUID="a9076853-d7e1-4260-ba51-0018d07b3928" pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:43.590501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.590472    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sbts\" (UniqueName: \"kubernetes.io/projected/a2dc72d3-39ea-4160-8e2c-7efeae1ccbde-kube-api-access-4sbts\") pod \"a2dc72d3-39ea-4160-8e2c-7efeae1ccbde\" (UID: \"a2dc72d3-39ea-4160-8e2c-7efeae1ccbde\") "
Sep 11 19:53:43.592455 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.592435    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dc72d3-39ea-4160-8e2c-7efeae1ccbde-kube-api-access-4sbts" (OuterVolumeSpecName: "kube-api-access-4sbts") pod "a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" (UID: "a2dc72d3-39ea-4160-8e2c-7efeae1ccbde"). InnerVolumeSpecName "kube-api-access-4sbts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:53:43.692180 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.692152    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sbts\" (UniqueName: \"kubernetes.io/projected/a2dc72d3-39ea-4160-8e2c-7efeae1ccbde-kube-api-access-4sbts\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:43.767256 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.767223    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:43.873158 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:43.873083    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7497-b324af\" is forbidden: User \"system:node:ip-10-0-136-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-27.ec2.internal' and this object" podUID="a9076853-d7e1-4260-ba51-0018d07b3928" pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:43.874252 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:43.874231    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" path="/var/lib/kubelet/pods/a2dc72d3-39ea-4160-8e2c-7efeae1ccbde/volumes"
Sep 11 19:53:44.372780 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:44.372739    2571 generic.go:356] "Generic (PLEG): container finished" podID="a2dc72d3-39ea-4160-8e2c-7efeae1ccbde" containerID="baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52" exitCode=137
Sep 11 19:53:44.373456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:44.372782    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-53c6fc"
Sep 11 19:53:44.373456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:44.372852    2571 scope.go:117] "RemoveContainer" containerID="baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52"
Sep 11 19:53:44.381792 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:44.381773    2571 scope.go:117] "RemoveContainer" containerID="baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52"
Sep 11 19:53:44.382109 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:44.382076    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52\": container with ID starting with baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52 not found: ID does not exist" containerID="baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52"
Sep 11 19:53:44.382206 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:44.382119    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52"} err="failed to get container status \"baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52\": rpc error: code = NotFound desc = could not find container \"baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52\": container with ID starting with baa9979b359dd361c0f94bb550e5d4e5de2f7d9f20a0c21c9cc43d2ceaa3fe52 not found: ID does not exist"
Sep 11 19:53:44.771907 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:44.771831    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:45.379519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:45.379490    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:45.380744 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:45.380725    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:53:45.534826 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:45.534804    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:45.608342 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:45.608316    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzt47\" (UniqueName: \"kubernetes.io/projected/a9076853-d7e1-4260-ba51-0018d07b3928-kube-api-access-mzt47\") pod \"a9076853-d7e1-4260-ba51-0018d07b3928\" (UID: \"a9076853-d7e1-4260-ba51-0018d07b3928\") "
Sep 11 19:53:45.610378 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:45.610348    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9076853-d7e1-4260-ba51-0018d07b3928-kube-api-access-mzt47" (OuterVolumeSpecName: "kube-api-access-mzt47") pod "a9076853-d7e1-4260-ba51-0018d07b3928" (UID: "a9076853-d7e1-4260-ba51-0018d07b3928"). InnerVolumeSpecName "kube-api-access-mzt47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:53:45.709161 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:45.709135    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzt47\" (UniqueName: \"kubernetes.io/projected/a9076853-d7e1-4260-ba51-0018d07b3928-kube-api-access-mzt47\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:45.873731 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:45.873700    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9076853-d7e1-4260-ba51-0018d07b3928" path="/var/lib/kubelet/pods/a9076853-d7e1-4260-ba51-0018d07b3928/volumes"
Sep 11 19:53:46.385929 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:46.385894    2571 generic.go:356] "Generic (PLEG): container finished" podID="a9076853-d7e1-4260-ba51-0018d07b3928" containerID="1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2" exitCode=137
Sep 11 19:53:46.386412 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:46.385939    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-b324af"
Sep 11 19:53:46.386412 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:46.385973    2571 scope.go:117] "RemoveContainer" containerID="1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2"
Sep 11 19:53:46.400869 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:46.400845    2571 scope.go:117] "RemoveContainer" containerID="1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2"
Sep 11 19:53:46.401189 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:46.401165    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2\": container with ID starting with 1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2 not found: ID does not exist" containerID="1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2"
Sep 11 19:53:46.401284 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:46.401199    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2"} err="failed to get container status \"1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2\": rpc error: code = NotFound desc = could not find container \"1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2\": container with ID starting with 1a608b0b6d6eab25685fa2380ed794078c8509e4c67491026e8f6bee2912f5a2 not found: ID does not exist"
Sep 11 19:53:46.994381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:46.994359    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:53:47.122703 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122679    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-data\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.122834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122737    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-token\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.122834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122758    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istiod-ca-cert\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.122834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122785    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-certs\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.122834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122812    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-credential-socket\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.123020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122840    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksfp8\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-kube-api-access-ksfp8\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.123020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122866    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-socket\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.123020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122888    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-envoy\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.123020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122911    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-podinfo\") pod \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\" (UID: \"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8\") "
Sep 11 19:53:47.123020 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.122945    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:47.123320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.123084    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:47.123320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.123125    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:47.123320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.123235    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:47.123320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.123279    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:53:47.123320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.123293    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.123320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.123304    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.123320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.123313    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.123320 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.123322    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.125011 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.124982    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:53:47.125140 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.125092    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:47.125229 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.125207    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-kube-api-access-ksfp8" (OuterVolumeSpecName: "kube-api-access-ksfp8") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "kube-api-access-ksfp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:53:47.125271 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.125240    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" (UID: "8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:53:47.224339 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.224313    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.224339 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.224335    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.224486 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.224346    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksfp8\" (UniqueName: \"kubernetes.io/projected/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-kube-api-access-ksfp8\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.224486 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.224355    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.224486 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.224363    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:47.393216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.393128    2571 generic.go:356] "Generic (PLEG): container finished" podID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerID="cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6" exitCode=0
Sep 11 19:53:47.393216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.393184    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"
Sep 11 19:53:47.393707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.393217    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" event={"ID":"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8","Type":"ContainerDied","Data":"cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6"}
Sep 11 19:53:47.393707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.393294    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh" event={"ID":"8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8","Type":"ContainerDied","Data":"cc3c799cfa5db27b79a02cfe8c718f37dfcc1348cca68e6cd8509a2c4f56a754"}
Sep 11 19:53:47.393707 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.393317    2571 scope.go:117] "RemoveContainer" containerID="cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6"
Sep 11 19:53:47.403049 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.403033    2571 scope.go:117] "RemoveContainer" containerID="d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e"
Sep 11 19:53:47.411458 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.411442    2571 scope.go:117] "RemoveContainer" containerID="cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6"
Sep 11 19:53:47.411726 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:47.411707    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6\": container with ID starting with cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6 not found: ID does not exist" containerID="cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6"
Sep 11 19:53:47.411773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.411734    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6"} err="failed to get container status \"cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6\": rpc error: code = NotFound desc = could not find container \"cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6\": container with ID starting with cb7dd09d2e6948ce8082be31ba609facd0ded62b99248fce43d738658bf045b6 not found: ID does not exist"
Sep 11 19:53:47.411773 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.411751    2571 scope.go:117] "RemoveContainer" containerID="d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e"
Sep 11 19:53:47.412023 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:53:47.412005    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e\": container with ID starting with d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e not found: ID does not exist" containerID="d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e"
Sep 11 19:53:47.412070 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.412028    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e"} err="failed to get container status \"d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e\": rpc error: code = NotFound desc = could not find container \"d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e\": container with ID starting with d99a3ca94caeb68250a383bed2e1e7ae305544a1f91c148704e4b709d9d3da4e not found: ID does not exist"
Sep 11 19:53:47.416985 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.416963    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"]
Sep 11 19:53:47.420210 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.420189    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-f76076-openshift-default-f7bb64d6d-kwwrh"]
Sep 11 19:53:47.873709 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:47.873679    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" path="/var/lib/kubelet/pods/8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8/volumes"
Sep 11 19:53:48.937241 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:48.937206    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"]
Sep 11 19:53:48.937583 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:48.937516    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy"
Sep 11 19:53:48.937583 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:48.937526    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy"
Sep 11 19:53:48.937717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:48.937597    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy"
Sep 11 19:53:48.937717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:48.937608    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy"
Sep 11 19:53:48.942116 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:48.942077    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:48.944968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:48.944948    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-105b25aa-openshift-default-dockercfg-vln4g\""
Sep 11 19:53:48.956835 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:48.956814    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"]
Sep 11 19:53:49.040734 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040709    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istiod-ca-cert\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.040850 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-credential-socket\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.040850 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040783    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-podinfo\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.040850 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrcg\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-kube-api-access-gvrcg\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.041003 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040851    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-socket\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.041003 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-token\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.041003 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040886    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-certs\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.041003 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-data\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.041003 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.040983    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-envoy\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142223 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142193    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-data\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142353 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-envoy\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142353 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istiod-ca-cert\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142353 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142348    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-credential-socket\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-podinfo\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrcg\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-kube-api-access-gvrcg\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-socket\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-token\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142719 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142628    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-data\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142719 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142675    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-certs\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142831 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142729    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-credential-socket\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142889 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142876    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-socket\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.142982 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.142958    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-certs\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.143180 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.143160    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istiod-ca-cert\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.145175 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.145150    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-envoy\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.145375 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.145356    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-podinfo\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.150683 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.150661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrcg\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-kube-api-access-gvrcg\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.151127 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.151084    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-token\") pod \"e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") " pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.252139 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.252045    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:49.533007 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.532981    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"]
Sep 11 19:53:49.533913 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:53:49.533879    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e0ac38_4c51_4bb9_b40d_4d30c367ce4f.slice/crio-2cc330973c5a39420d9c7ee9f9a129acd85452ed82b9b23de20b5c74690bbc0b WatchSource:0}: Error finding container 2cc330973c5a39420d9c7ee9f9a129acd85452ed82b9b23de20b5c74690bbc0b: Status 404 returned error can't find the container with id 2cc330973c5a39420d9c7ee9f9a129acd85452ed82b9b23de20b5c74690bbc0b
Sep 11 19:53:49.536213 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.536182    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:53:49.536288 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.536242    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:53:49.536288 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:49.536272    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:53:50.414169 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:50.414134    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" event={"ID":"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f","Type":"ContainerStarted","Data":"f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14"}
Sep 11 19:53:50.414555 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:50.414195    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" event={"ID":"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f","Type":"ContainerStarted","Data":"2cc330973c5a39420d9c7ee9f9a129acd85452ed82b9b23de20b5c74690bbc0b"}
Sep 11 19:53:50.437201 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:50.437151    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" podStartSLOduration=2.437135662 podStartE2EDuration="2.437135662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:50.434057415 +0000 UTC m=+1467.109994326" watchObservedRunningTime="2026-09-11 19:53:50.437135662 +0000 UTC m=+1467.113072571"
Sep 11 19:53:51.252816 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:51.252779    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:51.257756 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:51.257732    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:51.419924 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:51.419895    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:51.420837 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:51.420818    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:53:57.252548 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.252501    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd" podUID="4d1be39c-c770-4c09-80bb-83094f96345d" containerName="main" containerID="cri-o://f568fd1625044e5a1ea24e1515d7420d779f032f2385763274eec5d24130f0f0" gracePeriod=45
Sep 11 19:53:57.451686 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.451654    2571 generic.go:356] "Generic (PLEG): container finished" podID="4d1be39c-c770-4c09-80bb-83094f96345d" containerID="f568fd1625044e5a1ea24e1515d7420d779f032f2385763274eec5d24130f0f0" exitCode=0
Sep 11 19:53:57.451821 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.451732    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd" event={"ID":"4d1be39c-c770-4c09-80bb-83094f96345d","Type":"ContainerDied","Data":"f568fd1625044e5a1ea24e1515d7420d779f032f2385763274eec5d24130f0f0"}
Sep 11 19:53:57.496548 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.496520    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:53:57.618142 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618038    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-tmp-dir\") pod \"4d1be39c-c770-4c09-80bb-83094f96345d\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") "
Sep 11 19:53:57.618142 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618127    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d1be39c-c770-4c09-80bb-83094f96345d-tls-certs\") pod \"4d1be39c-c770-4c09-80bb-83094f96345d\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") "
Sep 11 19:53:57.618368 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618176    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-home\") pod \"4d1be39c-c770-4c09-80bb-83094f96345d\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") "
Sep 11 19:53:57.618368 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618305    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d1be39c-c770-4c09-80bb-83094f96345d" (UID: "4d1be39c-c770-4c09-80bb-83094f96345d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:57.618368 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618322    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-dshm\") pod \"4d1be39c-c770-4c09-80bb-83094f96345d\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") "
Sep 11 19:53:57.618523 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618390    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-model-cache\") pod \"4d1be39c-c770-4c09-80bb-83094f96345d\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") "
Sep 11 19:53:57.618523 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618407    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-home" (OuterVolumeSpecName: "home") pod "4d1be39c-c770-4c09-80bb-83094f96345d" (UID: "4d1be39c-c770-4c09-80bb-83094f96345d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:57.618523 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618437    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvfxm\" (UniqueName: \"kubernetes.io/projected/4d1be39c-c770-4c09-80bb-83094f96345d-kube-api-access-qvfxm\") pod \"4d1be39c-c770-4c09-80bb-83094f96345d\" (UID: \"4d1be39c-c770-4c09-80bb-83094f96345d\") "
Sep 11 19:53:57.618694 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618617    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d1be39c-c770-4c09-80bb-83094f96345d" (UID: "4d1be39c-c770-4c09-80bb-83094f96345d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:57.618758 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618724    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:57.618758 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618739    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:57.618758 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.618749    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:57.620376 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.620346    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-dshm" (OuterVolumeSpecName: "dshm") pod "4d1be39c-c770-4c09-80bb-83094f96345d" (UID: "4d1be39c-c770-4c09-80bb-83094f96345d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:53:57.620495 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.620387    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1be39c-c770-4c09-80bb-83094f96345d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d1be39c-c770-4c09-80bb-83094f96345d" (UID: "4d1be39c-c770-4c09-80bb-83094f96345d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:53:57.620495 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.620460    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1be39c-c770-4c09-80bb-83094f96345d-kube-api-access-qvfxm" (OuterVolumeSpecName: "kube-api-access-qvfxm") pod "4d1be39c-c770-4c09-80bb-83094f96345d" (UID: "4d1be39c-c770-4c09-80bb-83094f96345d"). InnerVolumeSpecName "kube-api-access-qvfxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:53:57.719138 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.719094    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d1be39c-c770-4c09-80bb-83094f96345d-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:57.719138 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.719135    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvfxm\" (UniqueName: \"kubernetes.io/projected/4d1be39c-c770-4c09-80bb-83094f96345d-kube-api-access-qvfxm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:57.719293 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:57.719146    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d1be39c-c770-4c09-80bb-83094f96345d-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:53:58.439960 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.439925    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"]
Sep 11 19:53:58.440421 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.440337    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1be39c-c770-4c09-80bb-83094f96345d" containerName="main"
Sep 11 19:53:58.440421 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.440352    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1be39c-c770-4c09-80bb-83094f96345d" containerName="main"
Sep 11 19:53:58.440421 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.440372    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy"
Sep 11 19:53:58.440421 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.440377    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8baf6a2e-cee6-4a4c-80c3-7ff12f2337f8" containerName="istio-proxy"
Sep 11 19:53:58.440552 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.440440    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1be39c-c770-4c09-80bb-83094f96345d" containerName="main"
Sep 11 19:53:58.443474 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.443438    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:58.446962 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.446940    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-105b25aa-dockercfg-k4cdh\""
Sep 11 19:53:58.447083 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.446941    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-105b25aa\""
Sep 11 19:53:58.456586 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.456565    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"]
Sep 11 19:53:58.459831 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.459808    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"
Sep 11 19:53:58.459965 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.459806    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd" event={"ID":"4d1be39c-c770-4c09-80bb-83094f96345d","Type":"ContainerDied","Data":"276ee748ea41feeb5873b248e62fe6f1b0f170bc7dc39bcb10844838e8e43436"}
Sep 11 19:53:58.459965 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.459904    2571 scope.go:117] "RemoveContainer" containerID="f568fd1625044e5a1ea24e1515d7420d779f032f2385763274eec5d24130f0f0"
Sep 11 19:53:58.488992 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.488964    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"]
Sep 11 19:53:58.494598 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.494575    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-04331c/test-model-04331c-kserve-5759cd96f7-8gxnd"]
Sep 11 19:53:58.526855 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.526813    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lsj5\" (UniqueName: \"kubernetes.io/projected/b9550029-c9ff-40b9-b12f-40260c9de2e6-kube-api-access-5lsj5\") pod \"payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr\" (UID: \"b9550029-c9ff-40b9-b12f-40260c9de2e6\") " pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:58.527004 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.526890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9550029-c9ff-40b9-b12f-40260c9de2e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr\" (UID: \"b9550029-c9ff-40b9-b12f-40260c9de2e6\") " pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:58.628063 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.628035    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lsj5\" (UniqueName: \"kubernetes.io/projected/b9550029-c9ff-40b9-b12f-40260c9de2e6-kube-api-access-5lsj5\") pod \"payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr\" (UID: \"b9550029-c9ff-40b9-b12f-40260c9de2e6\") " pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:58.628233 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.628089    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9550029-c9ff-40b9-b12f-40260c9de2e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr\" (UID: \"b9550029-c9ff-40b9-b12f-40260c9de2e6\") " pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:58.628833 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.628813    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9550029-c9ff-40b9-b12f-40260c9de2e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr\" (UID: \"b9550029-c9ff-40b9-b12f-40260c9de2e6\") " pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:58.636968 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.636936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lsj5\" (UniqueName: \"kubernetes.io/projected/b9550029-c9ff-40b9-b12f-40260c9de2e6-kube-api-access-5lsj5\") pod \"payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr\" (UID: \"b9550029-c9ff-40b9-b12f-40260c9de2e6\") " pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:58.754374 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:58.754307    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:59.032837 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:59.032317    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"]
Sep 11 19:53:59.033484 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:53:59.033454    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9550029_c9ff_40b9_b12f_40260c9de2e6.slice/crio-de7894163cac3d63c038b90084b63e7ae3c56dd2e3a41fdd1e2020f43d4a9974 WatchSource:0}: Error finding container de7894163cac3d63c038b90084b63e7ae3c56dd2e3a41fdd1e2020f43d4a9974: Status 404 returned error can't find the container with id de7894163cac3d63c038b90084b63e7ae3c56dd2e3a41fdd1e2020f43d4a9974
Sep 11 19:53:59.470754 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:59.470722    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr" event={"ID":"b9550029-c9ff-40b9-b12f-40260c9de2e6","Type":"ContainerStarted","Data":"cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0"}
Sep 11 19:53:59.471218 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:59.470782    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr" event={"ID":"b9550029-c9ff-40b9-b12f-40260c9de2e6","Type":"ContainerStarted","Data":"de7894163cac3d63c038b90084b63e7ae3c56dd2e3a41fdd1e2020f43d4a9974"}
Sep 11 19:53:59.471218 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:59.470814    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:53:59.491779 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:59.491726    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr" podStartSLOduration=1.491711131 podStartE2EDuration="1.491711131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:53:59.48954289 +0000 UTC m=+1476.165479811" watchObservedRunningTime="2026-09-11 19:53:59.491711131 +0000 UTC m=+1476.167648039"
Sep 11 19:53:59.875172 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:53:59.875141    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d1be39c-c770-4c09-80bb-83094f96345d" path="/var/lib/kubelet/pods/4d1be39c-c770-4c09-80bb-83094f96345d/volumes"
Sep 11 19:54:10.053612 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:10.053578    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:54:10.486069 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:10.486038    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:54:15.530501 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.530254    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"]
Sep 11 19:54:15.533330 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.530554    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p" podUID="4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" containerName="maas-api" containerID="cri-o://8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf" gracePeriod=30
Sep 11 19:54:15.822122 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.817591    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:54:15.880519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.880493    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k49vf\" (UniqueName: \"kubernetes.io/projected/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-kube-api-access-k49vf\") pod \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") "
Sep 11 19:54:15.880668 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.880533    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-metrics-certs\") pod \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") "
Sep 11 19:54:15.880668 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.880583    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-openshift-service-ca\") pod \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") "
Sep 11 19:54:15.880668 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.880608    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls\") pod \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\" (UID: \"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee\") "
Sep 11 19:54:15.880998 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.880961    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" (UID: "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:54:15.882744 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.882716    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-kube-api-access-k49vf" (OuterVolumeSpecName: "kube-api-access-k49vf") pod "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" (UID: "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee"). InnerVolumeSpecName "kube-api-access-k49vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:54:15.882845 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.882738    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" (UID: "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:54:15.882845 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.882764    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" (UID: "4c876d2f-b8d7-461a-8b57-ca2af6bb89ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:54:15.981455 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.981424    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-maas-api-tls\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:54:15.981455 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.981455    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k49vf\" (UniqueName: \"kubernetes.io/projected/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-kube-api-access-k49vf\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:54:15.981641 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.981464    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-metrics-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:54:15.981641 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:15.981473    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee-openshift-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:54:16.573883 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.573847    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"]
Sep 11 19:54:16.574391 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.574372    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" containerName="maas-api"
Sep 11 19:54:16.574458 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.574394    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" containerName="maas-api"
Sep 11 19:54:16.574527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.574481    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" containerName="maas-api"
Sep 11 19:54:16.575937 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.575906    2571 generic.go:356] "Generic (PLEG): container finished" podID="4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" containerID="8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf" exitCode=0
Sep 11 19:54:16.576087 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.576025    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"
Sep 11 19:54:16.577779 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.577756    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p" event={"ID":"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee","Type":"ContainerDied","Data":"8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf"}
Sep 11 19:54:16.577928 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.577790    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p" event={"ID":"4c876d2f-b8d7-461a-8b57-ca2af6bb89ee","Type":"ContainerDied","Data":"c47e59b302f4dcf3f4783a3123c62bf9e90c6b03f88118fabd37ccac3544e75c"}
Sep 11 19:54:16.577928 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.577812    2571 scope.go:117] "RemoveContainer" containerID="8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf"
Sep 11 19:54:16.577928 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.577872    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.581327 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.581310    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a8ee95\"/\"e2e-conflict-b-50f6e9f6-kserve-self-signed-certs\""
Sep 11 19:54:16.586621 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.586600    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-model-cache\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.586711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.586670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4psh\" (UniqueName: \"kubernetes.io/projected/4b33f972-6993-4f97-a76a-b23a5d8a23f5-kube-api-access-h4psh\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.586711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.586704    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-home\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.586780 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.586736    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tmp-dir\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.586874 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.586860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-dshm\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.586910 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.586883    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tls-certs\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.588662 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.588645    2571 scope.go:117] "RemoveContainer" containerID="8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf"
Sep 11 19:54:16.588929 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:54:16.588910    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf\": container with ID starting with 8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf not found: ID does not exist" containerID="8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf"
Sep 11 19:54:16.588992 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.588936    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf"} err="failed to get container status \"8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf\": rpc error: code = NotFound desc = could not find container \"8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf\": container with ID starting with 8d318d4f3590de0a3eec529d73014830f7243a2d2e73c91ca82dca7eb66a9edf not found: ID does not exist"
Sep 11 19:54:16.593916 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.593897    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"]
Sep 11 19:54:16.659918 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.659886    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"]
Sep 11 19:54:16.666172 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.666148    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-04331c-7db4f8b756-sxb2p"]
Sep 11 19:54:16.687540 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.687502    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-home\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.687540 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.687543    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tmp-dir\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.687667 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.687607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-dshm\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.687667 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.687634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tls-certs\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.687750 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.687688    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-model-cache\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.687750 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.687731    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4psh\" (UniqueName: \"kubernetes.io/projected/4b33f972-6993-4f97-a76a-b23a5d8a23f5-kube-api-access-h4psh\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.687980 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.687952    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-home\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.688079 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.688005    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tmp-dir\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.688159 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.688140    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-model-cache\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.689912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.689888    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-dshm\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.690066 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.690049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tls-certs\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.698425 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.698407    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4psh\" (UniqueName: \"kubernetes.io/projected/4b33f972-6993-4f97-a76a-b23a5d8a23f5-kube-api-access-h4psh\") pod \"e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") " pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:16.889736 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:16.889702    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:17.204522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:17.204235    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"]
Sep 11 19:54:17.583041 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:17.583002    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg" event={"ID":"4b33f972-6993-4f97-a76a-b23a5d8a23f5","Type":"ContainerStarted","Data":"80889dd23ae4c7720c0ddb7118eee8cf2f20491eeefffdc0da9e3145c82be79d"}
Sep 11 19:54:17.583496 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:17.583054    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg" event={"ID":"4b33f972-6993-4f97-a76a-b23a5d8a23f5","Type":"ContainerStarted","Data":"adc759c7a47668c797b2895da858db78649adf8f61e4c0ce56f99ce215961e58"}
Sep 11 19:54:17.873536 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:17.873503    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c876d2f-b8d7-461a-8b57-ca2af6bb89ee" path="/var/lib/kubelet/pods/4c876d2f-b8d7-461a-8b57-ca2af6bb89ee/volumes"
Sep 11 19:54:20.603627 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:20.603549    2571 generic.go:356] "Generic (PLEG): container finished" podID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerID="f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14" exitCode=0
Sep 11 19:54:20.603627 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:20.603609    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" event={"ID":"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f","Type":"ContainerDied","Data":"f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14"}
Sep 11 19:54:20.604042 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:20.604000    2571 scope.go:117] "RemoveContainer" containerID="f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14"
Sep 11 19:54:20.605257 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:20.605226    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:54:20.605333 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:20.605300    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:54:20.605369 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:20.605337    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:54:20.628573 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:20.628537    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg" podStartSLOduration=4.628522001 podStartE2EDuration="4.628522001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:17.605389248 +0000 UTC m=+1494.281326158" watchObservedRunningTime="2026-09-11 19:54:20.628522001 +0000 UTC m=+1497.304458908"
Sep 11 19:54:21.420571 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:21.420512    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:54:21.614503 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:21.614453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" event={"ID":"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f","Type":"ContainerStarted","Data":"6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d"}
Sep 11 19:54:22.252986 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:22.252951    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:54:22.254156 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:22.254126    2571 patch_prober.go:28] interesting pod/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:54:22.254284 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:22.254206    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 19:54:23.257852 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:23.257811    2571 patch_prober.go:28] interesting pod/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:54:23.258260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:23.257885    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 19:54:24.252472 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:24.252377    2571 patch_prober.go:28] interesting pod/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:54:24.252472 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:24.252430    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 19:54:25.253174 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:25.253138    2571 patch_prober.go:28] interesting pod/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:54:25.253537 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:25.253199    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 19:54:26.252596 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:26.252556    2571 patch_prober.go:28] interesting pod/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:54:26.252789 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:26.252625    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 11 19:54:26.890171 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:26.890138    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:26.890642 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:26.890235    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:26.902896 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:26.902874    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:27.256558 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:27.256486    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:54:27.256764 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:27.256742    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:54:27.257383 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:27.257367    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:54:27.664216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:27.664191    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:54:40.901468 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:40.901435    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:54:44.546933 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.546889    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"]
Sep 11 19:54:44.550294 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.550276    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.553503 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.553480    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b032e\"/\"distinct-2-w0-4b032e-kserve-self-signed-certs\""
Sep 11 19:54:44.564277 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.564258    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"]
Sep 11 19:54:44.634277 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.634248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-home\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.634371 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.634300    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-tmp-dir\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.634371 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.634349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-dshm\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.634442 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.634397    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e558950d-0027-4a14-9600-df9dbb6e86dc-tls-certs\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.634442 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.634418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-model-cache\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.634557 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.634485    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njc69\" (UniqueName: \"kubernetes.io/projected/e558950d-0027-4a14-9600-df9dbb6e86dc-kube-api-access-njc69\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.734969 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.734942    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njc69\" (UniqueName: \"kubernetes.io/projected/e558950d-0027-4a14-9600-df9dbb6e86dc-kube-api-access-njc69\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.735077 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.734992    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-home\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.735077 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.735048    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-tmp-dir\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.735235 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.735086    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-dshm\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.735235 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.735131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e558950d-0027-4a14-9600-df9dbb6e86dc-tls-certs\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.735235 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.735163    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-model-cache\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.735472 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.735449    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-tmp-dir\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.735472 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.735466    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-home\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.735585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.735525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-model-cache\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.737264 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.737241    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-dshm\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.737663 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.737638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e558950d-0027-4a14-9600-df9dbb6e86dc-tls-certs\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.747823 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.747805    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njc69\" (UniqueName: \"kubernetes.io/projected/e558950d-0027-4a14-9600-df9dbb6e86dc-kube-api-access-njc69\") pod \"distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") " pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:44.858645 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:44.858584    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:45.137789 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:45.137766    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"]
Sep 11 19:54:45.139078 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:54:45.139054    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode558950d_0027_4a14_9600_df9dbb6e86dc.slice/crio-43873b05c0bb52c4611f76ea625557a66b5a0e738bc7422e859a5ce7ef3dbbe1 WatchSource:0}: Error finding container 43873b05c0bb52c4611f76ea625557a66b5a0e738bc7422e859a5ce7ef3dbbe1: Status 404 returned error can't find the container with id 43873b05c0bb52c4611f76ea625557a66b5a0e738bc7422e859a5ce7ef3dbbe1
Sep 11 19:54:45.759331 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:45.759298    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr" event={"ID":"e558950d-0027-4a14-9600-df9dbb6e86dc","Type":"ContainerStarted","Data":"97441a64d791f01aad43d88ef8a93e4b10f617ef0bcf4d71729bcafd55ac34a9"}
Sep 11 19:54:45.759331 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:45.759337    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr" event={"ID":"e558950d-0027-4a14-9600-df9dbb6e86dc","Type":"ContainerStarted","Data":"43873b05c0bb52c4611f76ea625557a66b5a0e738bc7422e859a5ce7ef3dbbe1"}
Sep 11 19:54:45.777284 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:45.777238    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr" podStartSLOduration=1.77722384 podStartE2EDuration="1.77722384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:45.775374355 +0000 UTC m=+1522.451311274" watchObservedRunningTime="2026-09-11 19:54:45.77722384 +0000 UTC m=+1522.453160746"
Sep 11 19:54:54.859620 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:54.859588    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:54.860025 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:54.859631    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:54.871650 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:54.871623    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:54:55.822264 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:54:55.822235    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 19:55:06.416306 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.416273    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"]
Sep 11 19:55:06.419567 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.419547    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:06.422607 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.422582    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8a4eb7-dockercfg-nlc4s\""
Sep 11 19:55:06.422721 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.422591    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8a4eb7\""
Sep 11 19:55:06.431064 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.431041    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"]
Sep 11 19:55:06.515796 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.515761    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92j4\" (UniqueName: \"kubernetes.io/projected/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-kube-api-access-j92j4\") pod \"payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l\" (UID: \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:06.515967 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.515855    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l\" (UID: \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:06.617136 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.617082    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l\" (UID: \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:06.617265 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.617170    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92j4\" (UniqueName: \"kubernetes.io/projected/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-kube-api-access-j92j4\") pod \"payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l\" (UID: \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:06.617813 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.617792    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l\" (UID: \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:06.625038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.625017    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92j4\" (UniqueName: \"kubernetes.io/projected/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-kube-api-access-j92j4\") pod \"payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l\" (UID: \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:06.729893 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:06.729842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:07.003333 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.003309    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"]
Sep 11 19:55:07.004917 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:55:07.004894    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978af7f4_f1e8_427f_9fe0_777d92eb0ba4.slice/crio-6f5dae2d21419295af62bc6a385ea8e9769bccbe673d6a2033044dcee87150dd WatchSource:0}: Error finding container 6f5dae2d21419295af62bc6a385ea8e9769bccbe673d6a2033044dcee87150dd: Status 404 returned error can't find the container with id 6f5dae2d21419295af62bc6a385ea8e9769bccbe673d6a2033044dcee87150dd
Sep 11 19:55:07.731450 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.731415    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"]
Sep 11 19:55:07.734834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.734810    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.738875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.738847    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d46ae3\"/\"default-dockercfg-q8hzz\""
Sep 11 19:55:07.738875 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.738860    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d46ae3\"/\"kube-root-ca.crt\""
Sep 11 19:55:07.739057 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.738856    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d46ae3\"/\"noten-model-d46ae3-kserve-self-signed-certs\""
Sep 11 19:55:07.739057 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.738910    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d46ae3\"/\"openshift-service-ca.crt\""
Sep 11 19:55:07.750155 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.750137    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"]
Sep 11 19:55:07.826039 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.826011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-dshm\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.826171 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.826051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-model-cache\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.826171 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.826075    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-home\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.826171 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.826145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nds5l\" (UniqueName: \"kubernetes.io/projected/4a068842-aafb-450b-a091-6ec437f3842b-kube-api-access-nds5l\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.826286 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.826173    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a068842-aafb-450b-a091-6ec437f3842b-tls-certs\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.826286 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.826203    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-tmp-dir\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.874565 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.874534    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l" event={"ID":"978af7f4-f1e8-427f-9fe0-777d92eb0ba4","Type":"ContainerStarted","Data":"e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d"}
Sep 11 19:55:07.874687 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.874589    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l" event={"ID":"978af7f4-f1e8-427f-9fe0-777d92eb0ba4","Type":"ContainerStarted","Data":"6f5dae2d21419295af62bc6a385ea8e9769bccbe673d6a2033044dcee87150dd"}
Sep 11 19:55:07.874687 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.874619    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:07.892489 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.892439    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l" podStartSLOduration=1.892425078 podStartE2EDuration="1.892425078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:55:07.890860989 +0000 UTC m=+1544.566797909" watchObservedRunningTime="2026-09-11 19:55:07.892425078 +0000 UTC m=+1544.568361985"
Sep 11 19:55:07.927289 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927263    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-home\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.927381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927304    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nds5l\" (UniqueName: \"kubernetes.io/projected/4a068842-aafb-450b-a091-6ec437f3842b-kube-api-access-nds5l\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.927381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927326    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a068842-aafb-450b-a091-6ec437f3842b-tls-certs\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.927381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927364    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-tmp-dir\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.927533 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-dshm\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.927533 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927504    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-model-cache\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.927717 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927697    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-home\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.927876 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927851    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-tmp-dir\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.927961 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.927944    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-model-cache\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.929740 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.929721    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-dshm\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.929885 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.929868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a068842-aafb-450b-a091-6ec437f3842b-tls-certs\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:07.935390 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:07.935369    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nds5l\" (UniqueName: \"kubernetes.io/projected/4a068842-aafb-450b-a091-6ec437f3842b-kube-api-access-nds5l\") pod \"noten-model-d46ae3-kserve-5d67c4ccd7-q49w2\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") " pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:08.046462 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:08.046386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:08.320800 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:08.320755    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"]
Sep 11 19:55:08.884174 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:08.884080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2" event={"ID":"4a068842-aafb-450b-a091-6ec437f3842b","Type":"ContainerStarted","Data":"e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8"}
Sep 11 19:55:08.884174 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:08.884140    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2" event={"ID":"4a068842-aafb-450b-a091-6ec437f3842b","Type":"ContainerStarted","Data":"1079842fe3e8b943fb130c88f2666504708b63626a34c19870b884a41cf021d7"}
Sep 11 19:55:08.903394 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:08.903347    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2" podStartSLOduration=1.903333241 podStartE2EDuration="1.903333241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:55:08.901121383 +0000 UTC m=+1545.577058284" watchObservedRunningTime="2026-09-11 19:55:08.903333241 +0000 UTC m=+1545.579270148"
Sep 11 19:55:11.156340 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:11.156307    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:55:18.047434 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:18.047401    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:18.047897 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:18.047448    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:18.059655 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:18.059627    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:18.885915 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:18.885882    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 19:55:18.947043 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:18.947010    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:55:36.443877 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:36.443838    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:55:42.123317 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:42.123278    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"]
Sep 11 19:55:42.123711 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:42.123492    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5" podUID="66dd389d-11d6-4b26-abc9-688860246c31" containerName="istio-proxy" containerID="cri-o://110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129" gracePeriod=30
Sep 11 19:55:45.382494 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:45.382447    2571 patch_prober.go:28] interesting pod/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 19:55:45.382954 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:45.382517    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5" podUID="66dd389d-11d6-4b26-abc9-688860246c31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:55:47.365270 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.365247    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:55:47.412372 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.412300    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"]
Sep 11 19:55:47.502372 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502341    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66dd389d-11d6-4b26-abc9-688860246c31-istio-podinfo\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502551 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502381    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-data\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502551 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502405    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcstt\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-kube-api-access-jcstt\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502551 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502440    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-envoy\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502551 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502466    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-certs\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502551 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502517    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-istio-token\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502829 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502566    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-credential-socket\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502829 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502600    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66dd389d-11d6-4b26-abc9-688860246c31-istiod-ca-cert\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502829 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502630    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-socket\") pod \"66dd389d-11d6-4b26-abc9-688860246c31\" (UID: \"66dd389d-11d6-4b26-abc9-688860246c31\") "
Sep 11 19:55:47.502829 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502714    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-data" (OuterVolumeSpecName: "istio-data") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:55:47.502829 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502776    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:55:47.503094 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502948    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:55:47.503094 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.502976    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:55:47.503094 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.503043    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66dd389d-11d6-4b26-abc9-688860246c31-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:55:47.503094 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.503070    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:47.503094 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.503088    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:47.503408 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.503127    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:47.503408 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.503142    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:47.504582 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.504550    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:55:47.504582 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.504572    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-kube-api-access-jcstt" (OuterVolumeSpecName: "kube-api-access-jcstt") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "kube-api-access-jcstt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:55:47.504738 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.504595    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66dd389d-11d6-4b26-abc9-688860246c31-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:55:47.504826 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.504805    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-istio-token" (OuterVolumeSpecName: "istio-token") pod "66dd389d-11d6-4b26-abc9-688860246c31" (UID: "66dd389d-11d6-4b26-abc9-688860246c31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:55:47.604319 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.604280    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66dd389d-11d6-4b26-abc9-688860246c31-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:47.604319 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.604318    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcstt\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-kube-api-access-jcstt\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:47.604518 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.604331    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66dd389d-11d6-4b26-abc9-688860246c31-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:47.604518 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.604343    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66dd389d-11d6-4b26-abc9-688860246c31-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:47.604518 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:47.604354    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66dd389d-11d6-4b26-abc9-688860246c31-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:55:48.097260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.097223    2571 generic.go:356] "Generic (PLEG): container finished" podID="66dd389d-11d6-4b26-abc9-688860246c31" containerID="110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129" exitCode=0
Sep 11 19:55:48.097429 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.097279    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"
Sep 11 19:55:48.097429 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.097287    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5" event={"ID":"66dd389d-11d6-4b26-abc9-688860246c31","Type":"ContainerDied","Data":"110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129"}
Sep 11 19:55:48.097429 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.097341    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5" event={"ID":"66dd389d-11d6-4b26-abc9-688860246c31","Type":"ContainerDied","Data":"9bf1a536531a8bd4b07df0f32f6600bc9b62c9634a1e5d4df46b504c569baf78"}
Sep 11 19:55:48.097429 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.097371    2571 scope.go:117] "RemoveContainer" containerID="110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129"
Sep 11 19:55:48.106820 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.106806    2571 scope.go:117] "RemoveContainer" containerID="110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129"
Sep 11 19:55:48.107093 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:55:48.107069    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129\": container with ID starting with 110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129 not found: ID does not exist" containerID="110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129"
Sep 11 19:55:48.107178 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.107117    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129"} err="failed to get container status \"110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129\": rpc error: code = NotFound desc = could not find container \"110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129\": container with ID starting with 110dee1c90f656b3db68e62fa5ac2150d1cab6bb639a3545d703990730834129 not found: ID does not exist"
Sep 11 19:55:48.117886 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.117863    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"]
Sep 11 19:55:48.121336 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.121313    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d46ae3-openshift-default-5d8df4b5cd-cltm5"]
Sep 11 19:55:48.954142 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.954097    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"]
Sep 11 19:55:48.954635 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.954619    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66dd389d-11d6-4b26-abc9-688860246c31" containerName="istio-proxy"
Sep 11 19:55:48.954681 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.954638    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66dd389d-11d6-4b26-abc9-688860246c31" containerName="istio-proxy"
Sep 11 19:55:48.954742 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.954732    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="66dd389d-11d6-4b26-abc9-688860246c31" containerName="istio-proxy"
Sep 11 19:55:48.963585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.963560    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:48.967725 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.967701    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-105b25aa\"/\"default-dockercfg-hbt2f\""
Sep 11 19:55:48.967932 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.967902    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-105b25aa\"/\"kube-root-ca.crt\""
Sep 11 19:55:48.968135 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.968093    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-105b25aa\"/\"e2e-lifecycle-model-105b25aa-kserve-self-signed-certs\""
Sep 11 19:55:48.968332 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.968313    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-105b25aa\"/\"openshift-service-ca.crt\""
Sep 11 19:55:48.971158 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:48.971135    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"]
Sep 11 19:55:49.117826 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.117794    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-dshm\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.118065 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.117854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-model-cache\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.118065 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.117896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-home\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.118065 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.117926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tls-certs\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.118065 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.117947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tmp-dir\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.118065 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.117994    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/aba22e26-b6ef-43b8-9b88-a21e811bdf25-kube-api-access-w8h4h\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219322 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219252    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-dshm\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219322 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219304    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-model-cache\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219515 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-home\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219515 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219365    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tls-certs\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219515 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219392    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tmp-dir\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219515 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219440    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/aba22e26-b6ef-43b8-9b88-a21e811bdf25-kube-api-access-w8h4h\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219746    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-home\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-model-cache\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.219834 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.219822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tmp-dir\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.221491 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.221466    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-dshm\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.221765 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.221752    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tls-certs\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.227735 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.227718    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/aba22e26-b6ef-43b8-9b88-a21e811bdf25-kube-api-access-w8h4h\") pod \"e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") " pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.274263 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.274243    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:49.547333 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.547303    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"]
Sep 11 19:55:49.548389 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:55:49.548365    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaba22e26_b6ef_43b8_9b88_a21e811bdf25.slice/crio-65be374620688b3bec1cc0cc0cd3e3931821a3fc5bbc8be44267d4f1b94fc35d WatchSource:0}: Error finding container 65be374620688b3bec1cc0cc0cd3e3931821a3fc5bbc8be44267d4f1b94fc35d: Status 404 returned error can't find the container with id 65be374620688b3bec1cc0cc0cd3e3931821a3fc5bbc8be44267d4f1b94fc35d
Sep 11 19:55:49.874038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:49.874013    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66dd389d-11d6-4b26-abc9-688860246c31" path="/var/lib/kubelet/pods/66dd389d-11d6-4b26-abc9-688860246c31/volumes"
Sep 11 19:55:50.110937 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:50.110902    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm" event={"ID":"aba22e26-b6ef-43b8-9b88-a21e811bdf25","Type":"ContainerStarted","Data":"4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24"}
Sep 11 19:55:50.111399 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:50.110952    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm" event={"ID":"aba22e26-b6ef-43b8-9b88-a21e811bdf25","Type":"ContainerStarted","Data":"65be374620688b3bec1cc0cc0cd3e3931821a3fc5bbc8be44267d4f1b94fc35d"}
Sep 11 19:55:50.128835 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:50.128755    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm" podStartSLOduration=2.128739281 podStartE2EDuration="2.128739281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:55:50.126525015 +0000 UTC m=+1586.802461936" watchObservedRunningTime="2026-09-11 19:55:50.128739281 +0000 UTC m=+1586.804676188"
Sep 11 19:55:59.275317 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:59.275279    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:59.275774 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:59.275369    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:55:59.287880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:55:59.287853    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:56:00.179480 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:00.179445    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:56:02.431179 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.431141    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2" podUID="4a068842-aafb-450b-a091-6ec437f3842b" containerName="main" containerID="cri-o://e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8" gracePeriod=45
Sep 11 19:56:02.678046 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.678021    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:56:02.753118 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753021    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-model-cache\") pod \"4a068842-aafb-450b-a091-6ec437f3842b\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") "
Sep 11 19:56:02.753118 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753063    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a068842-aafb-450b-a091-6ec437f3842b-tls-certs\") pod \"4a068842-aafb-450b-a091-6ec437f3842b\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") "
Sep 11 19:56:02.753348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753143    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nds5l\" (UniqueName: \"kubernetes.io/projected/4a068842-aafb-450b-a091-6ec437f3842b-kube-api-access-nds5l\") pod \"4a068842-aafb-450b-a091-6ec437f3842b\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") "
Sep 11 19:56:02.753348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753285    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-home\") pod \"4a068842-aafb-450b-a091-6ec437f3842b\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") "
Sep 11 19:56:02.753348 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753339    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-dshm\") pod \"4a068842-aafb-450b-a091-6ec437f3842b\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") "
Sep 11 19:56:02.753527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753343    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a068842-aafb-450b-a091-6ec437f3842b" (UID: "4a068842-aafb-450b-a091-6ec437f3842b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:56:02.753527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753415    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-tmp-dir\") pod \"4a068842-aafb-450b-a091-6ec437f3842b\" (UID: \"4a068842-aafb-450b-a091-6ec437f3842b\") "
Sep 11 19:56:02.753527 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753491    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-home" (OuterVolumeSpecName: "home") pod "4a068842-aafb-450b-a091-6ec437f3842b" (UID: "4a068842-aafb-450b-a091-6ec437f3842b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:56:02.753751 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753729    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a068842-aafb-450b-a091-6ec437f3842b" (UID: "4a068842-aafb-450b-a091-6ec437f3842b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:56:02.753822 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753802    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:56:02.753879 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.753825    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:56:02.755386 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.755361    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a068842-aafb-450b-a091-6ec437f3842b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a068842-aafb-450b-a091-6ec437f3842b" (UID: "4a068842-aafb-450b-a091-6ec437f3842b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:56:02.755602 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.755576    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-dshm" (OuterVolumeSpecName: "dshm") pod "4a068842-aafb-450b-a091-6ec437f3842b" (UID: "4a068842-aafb-450b-a091-6ec437f3842b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:56:02.755684 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.755599    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a068842-aafb-450b-a091-6ec437f3842b-kube-api-access-nds5l" (OuterVolumeSpecName: "kube-api-access-nds5l") pod "4a068842-aafb-450b-a091-6ec437f3842b" (UID: "4a068842-aafb-450b-a091-6ec437f3842b"). InnerVolumeSpecName "kube-api-access-nds5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:56:02.854355 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.854322    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nds5l\" (UniqueName: \"kubernetes.io/projected/4a068842-aafb-450b-a091-6ec437f3842b-kube-api-access-nds5l\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:56:02.854355 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.854350    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:56:02.854355 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.854361    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a068842-aafb-450b-a091-6ec437f3842b-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:56:02.854596 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:02.854368    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a068842-aafb-450b-a091-6ec437f3842b-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:56:03.185500 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.185468    2571 generic.go:356] "Generic (PLEG): container finished" podID="4a068842-aafb-450b-a091-6ec437f3842b" containerID="e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8" exitCode=0
Sep 11 19:56:03.185680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.185525    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2" event={"ID":"4a068842-aafb-450b-a091-6ec437f3842b","Type":"ContainerDied","Data":"e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8"}
Sep 11 19:56:03.185680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.185556    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2" event={"ID":"4a068842-aafb-450b-a091-6ec437f3842b","Type":"ContainerDied","Data":"1079842fe3e8b943fb130c88f2666504708b63626a34c19870b884a41cf021d7"}
Sep 11 19:56:03.185680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.185572    2571 scope.go:117] "RemoveContainer" containerID="e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8"
Sep 11 19:56:03.185680 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.185528    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"
Sep 11 19:56:03.195038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.195016    2571 scope.go:117] "RemoveContainer" containerID="e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8"
Sep 11 19:56:03.195328 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:56:03.195307    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8\": container with ID starting with e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8 not found: ID does not exist" containerID="e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8"
Sep 11 19:56:03.195387 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.195337    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8"} err="failed to get container status \"e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8\": rpc error: code = NotFound desc = could not find container \"e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8\": container with ID starting with e8f2e2937823d5b18c50b3e733d2daf61bebc6aec80d31a02aab9b4592518df8 not found: ID does not exist"
Sep 11 19:56:03.207804 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.207781    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"]
Sep 11 19:56:03.211481 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.211459    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d46ae3/noten-model-d46ae3-kserve-5d67c4ccd7-q49w2"]
Sep 11 19:56:03.874007 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:03.873973    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a068842-aafb-450b-a091-6ec437f3842b" path="/var/lib/kubelet/pods/4a068842-aafb-450b-a091-6ec437f3842b/volumes"
Sep 11 19:56:05.548492 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:05.548451    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:56:19.649457 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.649378    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"]
Sep 11 19:56:19.649823 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.649752    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a068842-aafb-450b-a091-6ec437f3842b" containerName="main"
Sep 11 19:56:19.649823 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.649764    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a068842-aafb-450b-a091-6ec437f3842b" containerName="main"
Sep 11 19:56:19.649901 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.649844    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a068842-aafb-450b-a091-6ec437f3842b" containerName="main"
Sep 11 19:56:19.654072 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.654054    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.658176 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.658141    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"openshift-service-ca.crt\""
Sep 11 19:56:19.658303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.658176    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"auth-test-model-57d97d-kserve-self-signed-certs\""
Sep 11 19:56:19.658303 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.658181    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"kube-root-ca.crt\""
Sep 11 19:56:19.658424 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.658350    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"default-dockercfg-s2gvw\""
Sep 11 19:56:19.661551 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.661527    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"]
Sep 11 19:56:19.804844 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.804809    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-tmp-dir\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.805056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.804854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-dshm\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.805056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.804875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n75t\" (UniqueName: \"kubernetes.io/projected/9dd62292-7ddd-4381-91de-32610918b7c7-kube-api-access-9n75t\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.805056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.804922    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-home\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.805056 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.804943    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd62292-7ddd-4381-91de-32610918b7c7-tls-certs\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.805219 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.805047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-model-cache\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906295    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-model-cache\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906366 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-tmp-dir\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906396    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-dshm\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906421    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n75t\" (UniqueName: \"kubernetes.io/projected/9dd62292-7ddd-4381-91de-32610918b7c7-kube-api-access-9n75t\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-home\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906572 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906455    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd62292-7ddd-4381-91de-32610918b7c7-tls-certs\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906777 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906749    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-model-cache\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906838 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906774    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-tmp-dir\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.906895 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.906851    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-home\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.908701 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.908675    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-dshm\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.908979 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.908961    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd62292-7ddd-4381-91de-32610918b7c7-tls-certs\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.914216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.914193    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n75t\" (UniqueName: \"kubernetes.io/projected/9dd62292-7ddd-4381-91de-32610918b7c7-kube-api-access-9n75t\") pod \"auth-test-model-57d97d-kserve-559d9d9d99-9r7zc\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:19.964246 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:19.964223    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:20.233281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:20.233257    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"]
Sep 11 19:56:20.237749 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:56:20.237701    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dd62292_7ddd_4381_91de_32610918b7c7.slice/crio-6bbbfcfcab7b22d9fb1891d0de28c215095eaf0eb3a83acdb39f172e391eea1c WatchSource:0}: Error finding container 6bbbfcfcab7b22d9fb1891d0de28c215095eaf0eb3a83acdb39f172e391eea1c: Status 404 returned error can't find the container with id 6bbbfcfcab7b22d9fb1891d0de28c215095eaf0eb3a83acdb39f172e391eea1c
Sep 11 19:56:20.279898 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:20.279866    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc" event={"ID":"9dd62292-7ddd-4381-91de-32610918b7c7","Type":"ContainerStarted","Data":"6bbbfcfcab7b22d9fb1891d0de28c215095eaf0eb3a83acdb39f172e391eea1c"}
Sep 11 19:56:21.287315 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:21.287281    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc" event={"ID":"9dd62292-7ddd-4381-91de-32610918b7c7","Type":"ContainerStarted","Data":"ab2a4cf5bb7fa465df05382ac82dff790f3f0d4eadfce78799f345bbeb1daf9b"}
Sep 11 19:56:21.313965 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:21.313913    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc" podStartSLOduration=2.3138989309999998 podStartE2EDuration="2.313898931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:56:21.312084868 +0000 UTC m=+1617.988021787" watchObservedRunningTime="2026-09-11 19:56:21.313898931 +0000 UTC m=+1617.989835838"
Sep 11 19:56:29.964685 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:29.964655    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:29.965095 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:29.964888    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:29.977165 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:29.977145    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:30.344531 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:30.344454    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 19:56:37.854831 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:56:37.854801    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:57:01.345769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:01.345733    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"]
Sep 11 19:57:06.369987 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:06.369955    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"]
Sep 11 19:57:16.365094 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.365050    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg" podUID="4b33f972-6993-4f97-a76a-b23a5d8a23f5" containerName="main" containerID="cri-o://80889dd23ae4c7720c0ddb7118eee8cf2f20491eeefffdc0da9e3145c82be79d" gracePeriod=45
Sep 11 19:57:16.579987 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.579954    2571 generic.go:356] "Generic (PLEG): container finished" podID="4b33f972-6993-4f97-a76a-b23a5d8a23f5" containerID="80889dd23ae4c7720c0ddb7118eee8cf2f20491eeefffdc0da9e3145c82be79d" exitCode=0
Sep 11 19:57:16.580156 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.580025    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg" event={"ID":"4b33f972-6993-4f97-a76a-b23a5d8a23f5","Type":"ContainerDied","Data":"80889dd23ae4c7720c0ddb7118eee8cf2f20491eeefffdc0da9e3145c82be79d"}
Sep 11 19:57:16.625199 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.625173    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:57:16.795211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795172    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4psh\" (UniqueName: \"kubernetes.io/projected/4b33f972-6993-4f97-a76a-b23a5d8a23f5-kube-api-access-h4psh\") pod \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") "
Sep 11 19:57:16.795401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795268    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-dshm\") pod \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") "
Sep 11 19:57:16.795401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795310    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-model-cache\") pod \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") "
Sep 11 19:57:16.795401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795328    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tmp-dir\") pod \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") "
Sep 11 19:57:16.795401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795347    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tls-certs\") pod \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") "
Sep 11 19:57:16.795401 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795398    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-home\") pod \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\" (UID: \"4b33f972-6993-4f97-a76a-b23a5d8a23f5\") "
Sep 11 19:57:16.795672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795580    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b33f972-6993-4f97-a76a-b23a5d8a23f5" (UID: "4b33f972-6993-4f97-a76a-b23a5d8a23f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:57:16.795672 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795597    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b33f972-6993-4f97-a76a-b23a5d8a23f5" (UID: "4b33f972-6993-4f97-a76a-b23a5d8a23f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:57:16.795774 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795706    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:16.795774 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795743    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:16.795851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.795781    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-home" (OuterVolumeSpecName: "home") pod "4b33f972-6993-4f97-a76a-b23a5d8a23f5" (UID: "4b33f972-6993-4f97-a76a-b23a5d8a23f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:57:16.797512 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.797487    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-dshm" (OuterVolumeSpecName: "dshm") pod "4b33f972-6993-4f97-a76a-b23a5d8a23f5" (UID: "4b33f972-6993-4f97-a76a-b23a5d8a23f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:57:16.797628 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.797506    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b33f972-6993-4f97-a76a-b23a5d8a23f5" (UID: "4b33f972-6993-4f97-a76a-b23a5d8a23f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:57:16.797628 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.797522    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b33f972-6993-4f97-a76a-b23a5d8a23f5-kube-api-access-h4psh" (OuterVolumeSpecName: "kube-api-access-h4psh") pod "4b33f972-6993-4f97-a76a-b23a5d8a23f5" (UID: "4b33f972-6993-4f97-a76a-b23a5d8a23f5"). InnerVolumeSpecName "kube-api-access-h4psh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:57:16.897092 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.897024    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33f972-6993-4f97-a76a-b23a5d8a23f5-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:16.897092 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.897049    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:16.897092 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.897058    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4psh\" (UniqueName: \"kubernetes.io/projected/4b33f972-6993-4f97-a76a-b23a5d8a23f5-kube-api-access-h4psh\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:16.897092 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:16.897067    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33f972-6993-4f97-a76a-b23a5d8a23f5-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:17.586609 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:17.586579    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"
Sep 11 19:57:17.587017 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:17.586578    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg" event={"ID":"4b33f972-6993-4f97-a76a-b23a5d8a23f5","Type":"ContainerDied","Data":"adc759c7a47668c797b2895da858db78649adf8f61e4c0ce56f99ce215961e58"}
Sep 11 19:57:17.587017 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:17.586704    2571 scope.go:117] "RemoveContainer" containerID="80889dd23ae4c7720c0ddb7118eee8cf2f20491eeefffdc0da9e3145c82be79d"
Sep 11 19:57:17.608659 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:17.608635    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"]
Sep 11 19:57:17.612964 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:17.612941    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-b-50f6e9f6-kserve-6576466794-5lnkg"]
Sep 11 19:57:17.873481 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:17.873458    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b33f972-6993-4f97-a76a-b23a5d8a23f5" path="/var/lib/kubelet/pods/4b33f972-6993-4f97-a76a-b23a5d8a23f5/volumes"
Sep 11 19:57:18.346769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:18.346687    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:57:21.388422 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.388385    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq" podUID="e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" containerName="main" containerID="cri-o://4832aa3ff6613bccb9f7e63fad272c3feeb2d561eaf87f57c322a4efb3dced77" gracePeriod=45
Sep 11 19:57:21.610704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.610673    2571 generic.go:356] "Generic (PLEG): container finished" podID="e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" containerID="4832aa3ff6613bccb9f7e63fad272c3feeb2d561eaf87f57c322a4efb3dced77" exitCode=0
Sep 11 19:57:21.610847 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.610731    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq" event={"ID":"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0","Type":"ContainerDied","Data":"4832aa3ff6613bccb9f7e63fad272c3feeb2d561eaf87f57c322a4efb3dced77"}
Sep 11 19:57:21.629788 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.629764    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:57:21.638016 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.637995    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-home\") pod \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") "
Sep 11 19:57:21.638158 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638048    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tmp-dir\") pod \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") "
Sep 11 19:57:21.638158 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638087    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-model-cache\") pod \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") "
Sep 11 19:57:21.638274 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638156    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktfzr\" (UniqueName: \"kubernetes.io/projected/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-kube-api-access-ktfzr\") pod \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") "
Sep 11 19:57:21.638274 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638259    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-dshm\") pod \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") "
Sep 11 19:57:21.638387 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638274    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" (UID: "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:57:21.638387 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638291    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-home" (OuterVolumeSpecName: "home") pod "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" (UID: "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:57:21.638387 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638310    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tls-certs\") pod \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\" (UID: \"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0\") "
Sep 11 19:57:21.638387 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638312    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" (UID: "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:57:21.638655 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638611    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:21.638655 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638630    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:21.638655 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.638642    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:21.640075 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.640057    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-dshm" (OuterVolumeSpecName: "dshm") pod "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" (UID: "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:57:21.640198 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.640178    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" (UID: "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:57:21.640246 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.640231    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-kube-api-access-ktfzr" (OuterVolumeSpecName: "kube-api-access-ktfzr") pod "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" (UID: "e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0"). InnerVolumeSpecName "kube-api-access-ktfzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:57:21.739067 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.739025    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:21.739067 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.739058    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktfzr\" (UniqueName: \"kubernetes.io/projected/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-kube-api-access-ktfzr\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:21.739067 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:21.739071    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:22.617652 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:22.617620    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"
Sep 11 19:57:22.617652 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:22.617638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq" event={"ID":"e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0","Type":"ContainerDied","Data":"7f63781c2ee3ba5b4ce11916d06aeb11519c5c0eac01a983b24be61f67bc277f"}
Sep 11 19:57:22.618174 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:22.617687    2571 scope.go:117] "RemoveContainer" containerID="4832aa3ff6613bccb9f7e63fad272c3feeb2d561eaf87f57c322a4efb3dced77"
Sep 11 19:57:22.637187 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:22.637159    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"]
Sep 11 19:57:22.642224 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:22.642200    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/e2e-conflict-a-50f6e9f6-kserve-78786998f6-9spjq"]
Sep 11 19:57:23.875373 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:23.875336    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" path="/var/lib/kubelet/pods/e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0/volumes"
Sep 11 19:57:25.573169 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.573132    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"]
Sep 11 19:57:25.573531 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.573473    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" containerName="main"
Sep 11 19:57:25.573531 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.573484    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" containerName="main"
Sep 11 19:57:25.573531 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.573502    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b33f972-6993-4f97-a76a-b23a5d8a23f5" containerName="main"
Sep 11 19:57:25.573531 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.573507    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b33f972-6993-4f97-a76a-b23a5d8a23f5" containerName="main"
Sep 11 19:57:25.573691 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.573579    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b33f972-6993-4f97-a76a-b23a5d8a23f5" containerName="main"
Sep 11 19:57:25.573691 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.573588    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6212a4e-4f6d-4173-a6d7-18fb8c8dfee0" containerName="main"
Sep 11 19:57:25.576519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.576503    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"
Sep 11 19:57:25.579184 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.579159    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:57:25.579314 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.579206    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 19:57:25.584585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.584558    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"]
Sep 11 19:57:25.671464 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.671415    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69051be5-856f-44bb-a5ab-86f5340aadaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc\" (UID: \"69051be5-856f-44bb-a5ab-86f5340aadaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"
Sep 11 19:57:25.772874 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.772836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69051be5-856f-44bb-a5ab-86f5340aadaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc\" (UID: \"69051be5-856f-44bb-a5ab-86f5340aadaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"
Sep 11 19:57:25.773549 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.773527    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69051be5-856f-44bb-a5ab-86f5340aadaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc\" (UID: \"69051be5-856f-44bb-a5ab-86f5340aadaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"
Sep 11 19:57:25.887851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:25.887803    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"
Sep 11 19:57:26.161211 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.161182    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"]
Sep 11 19:57:26.162804 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:57:26.162775    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69051be5_856f_44bb_a5ab_86f5340aadaf.slice/crio-c4f1fe8f227d4e539c94f45108f2a6ad79852aca72fd8974e2d45f7ec78d833a WatchSource:0}: Error finding container c4f1fe8f227d4e539c94f45108f2a6ad79852aca72fd8974e2d45f7ec78d833a: Status 404 returned error can't find the container with id c4f1fe8f227d4e539c94f45108f2a6ad79852aca72fd8974e2d45f7ec78d833a
Sep 11 19:57:26.642770 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.642727    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc" event={"ID":"69051be5-856f-44bb-a5ab-86f5340aadaf","Type":"ContainerStarted","Data":"7bdadaaf55a74cd3b12e63c918bede43d727e9ef73e86e6bd07728d9ba7c180f"}
Sep 11 19:57:26.643166 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.642798    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc" event={"ID":"69051be5-856f-44bb-a5ab-86f5340aadaf","Type":"ContainerStarted","Data":"c4f1fe8f227d4e539c94f45108f2a6ad79852aca72fd8974e2d45f7ec78d833a"}
Sep 11 19:57:26.660991 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.660944    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc" podStartSLOduration=1.6609257 podStartE2EDuration="1.6609257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:57:26.658794566 +0000 UTC m=+1683.334731473" watchObservedRunningTime="2026-09-11 19:57:26.6609257 +0000 UTC m=+1683.336862609"
Sep 11 19:57:26.750253 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.750221    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"]
Sep 11 19:57:26.753733 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.753714    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.756284 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.756262    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a8ee95\"/\"distinct-w4-a8ee95-kserve-self-signed-certs\""
Sep 11 19:57:26.763795 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.763772    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"]
Sep 11 19:57:26.782667 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.782638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993a257-1071-4519-bc3b-4ae5f9216ebd-tls-certs\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.782760 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.782677    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-model-cache\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.782760 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.782702    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ns2\" (UniqueName: \"kubernetes.io/projected/2993a257-1071-4519-bc3b-4ae5f9216ebd-kube-api-access-w4ns2\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.782840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.782766    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-tmp-dir\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.782840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.782801    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-dshm\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.782840 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.782832    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-home\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.883597 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.883571    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-tmp-dir\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.883745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.883622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-dshm\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.883745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.883640    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-home\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.883745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.883682    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993a257-1071-4519-bc3b-4ae5f9216ebd-tls-certs\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.883745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.883707    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-model-cache\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.883745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.883740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ns2\" (UniqueName: \"kubernetes.io/projected/2993a257-1071-4519-bc3b-4ae5f9216ebd-kube-api-access-w4ns2\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.884049 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.884008    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-tmp-dir\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.884129 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.884077    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-home\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.884201 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.884177    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-model-cache\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.885858 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.885838    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-dshm\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.886049 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.886031    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993a257-1071-4519-bc3b-4ae5f9216ebd-tls-certs\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:26.894284 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:26.894236    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ns2\" (UniqueName: \"kubernetes.io/projected/2993a257-1071-4519-bc3b-4ae5f9216ebd-kube-api-access-w4ns2\") pod \"distinct-w4-a8ee95-kserve-b548f7498-mcv44\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") " pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:27.063498 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:27.063458    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:27.336890 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:27.336863    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"]
Sep 11 19:57:27.339692 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:57:27.339657    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2993a257_1071_4519_bc3b_4ae5f9216ebd.slice/crio-46c6c74eb5cefbbbd0e8f4ae6d73331efc549c9890b0c0868f518f7043755d73 WatchSource:0}: Error finding container 46c6c74eb5cefbbbd0e8f4ae6d73331efc549c9890b0c0868f518f7043755d73: Status 404 returned error can't find the container with id 46c6c74eb5cefbbbd0e8f4ae6d73331efc549c9890b0c0868f518f7043755d73
Sep 11 19:57:27.650138 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:27.650084    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44" event={"ID":"2993a257-1071-4519-bc3b-4ae5f9216ebd","Type":"ContainerStarted","Data":"65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec"}
Sep 11 19:57:27.650585 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:27.650150    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44" event={"ID":"2993a257-1071-4519-bc3b-4ae5f9216ebd","Type":"ContainerStarted","Data":"46c6c74eb5cefbbbd0e8f4ae6d73331efc549c9890b0c0868f518f7043755d73"}
Sep 11 19:57:27.651575 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:27.651551    2571 generic.go:356] "Generic (PLEG): container finished" podID="69051be5-856f-44bb-a5ab-86f5340aadaf" containerID="7bdadaaf55a74cd3b12e63c918bede43d727e9ef73e86e6bd07728d9ba7c180f" exitCode=0
Sep 11 19:57:27.651704 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:27.651630    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc" event={"ID":"69051be5-856f-44bb-a5ab-86f5340aadaf","Type":"ContainerDied","Data":"7bdadaaf55a74cd3b12e63c918bede43d727e9ef73e86e6bd07728d9ba7c180f"}
Sep 11 19:57:27.667411 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:27.667372    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44" podStartSLOduration=1.6673587909999998 podStartE2EDuration="1.667358791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:57:27.666143888 +0000 UTC m=+1684.342080797" watchObservedRunningTime="2026-09-11 19:57:27.667358791 +0000 UTC m=+1684.343295699"
Sep 11 19:57:28.785744 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:28.785721    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"
Sep 11 19:57:28.802962 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:28.802861    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69051be5-856f-44bb-a5ab-86f5340aadaf-maas-api-service-ca\") pod \"69051be5-856f-44bb-a5ab-86f5340aadaf\" (UID: \"69051be5-856f-44bb-a5ab-86f5340aadaf\") "
Sep 11 19:57:28.803281 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:28.803250    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69051be5-856f-44bb-a5ab-86f5340aadaf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69051be5-856f-44bb-a5ab-86f5340aadaf" (UID: "69051be5-856f-44bb-a5ab-86f5340aadaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:57:28.803908 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:28.803347    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69051be5-856f-44bb-a5ab-86f5340aadaf-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:29.663981 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:29.663944    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc" event={"ID":"69051be5-856f-44bb-a5ab-86f5340aadaf","Type":"ContainerDied","Data":"c4f1fe8f227d4e539c94f45108f2a6ad79852aca72fd8974e2d45f7ec78d833a"}
Sep 11 19:57:29.663981 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:29.663979    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"
Sep 11 19:57:29.664216 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:29.664002    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4f1fe8f227d4e539c94f45108f2a6ad79852aca72fd8974e2d45f7ec78d833a"
Sep 11 19:57:32.255166 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.255128    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"]
Sep 11 19:57:32.255539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.255478    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69051be5-856f-44bb-a5ab-86f5340aadaf" containerName="revoke-keys"
Sep 11 19:57:32.255539 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.255489    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="69051be5-856f-44bb-a5ab-86f5340aadaf" containerName="revoke-keys"
Sep 11 19:57:32.255636 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.255545    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="69051be5-856f-44bb-a5ab-86f5340aadaf" containerName="revoke-keys"
Sep 11 19:57:32.258522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.258503    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.261309 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.261286    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"kube-root-ca.crt\""
Sep 11 19:57:32.262649 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.262619    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"auth-test-model-8a4eb7-kserve-self-signed-certs\""
Sep 11 19:57:32.262763 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.262653    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"openshift-service-ca.crt\""
Sep 11 19:57:32.262763 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.262653    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"default-dockercfg-v47l5\""
Sep 11 19:57:32.269090 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.269067    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"]
Sep 11 19:57:32.341405 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.341373    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-home\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.341596 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.341434    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-dshm\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.341596 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.341476    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77bc4a78-b956-45a2-b117-fe4efac876a7-tls-certs\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.341596 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.341514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qvj\" (UniqueName: \"kubernetes.io/projected/77bc4a78-b956-45a2-b117-fe4efac876a7-kube-api-access-79qvj\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.341596 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.341587    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-tmp-dir\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.341769 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.341623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-model-cache\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.442690 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.442653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-tmp-dir\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.442871 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.442701    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-model-cache\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.442871 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.442746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-home\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.442991 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.442888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-dshm\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.442991 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.442927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77bc4a78-b956-45a2-b117-fe4efac876a7-tls-certs\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.443141 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.443052    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qvj\" (UniqueName: \"kubernetes.io/projected/77bc4a78-b956-45a2-b117-fe4efac876a7-kube-api-access-79qvj\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.443204 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.443159    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-tmp-dir\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.443204 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.443183    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-home\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.443286 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.443211    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-model-cache\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.445215 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.445188    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-dshm\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.445383 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.445368    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77bc4a78-b956-45a2-b117-fe4efac876a7-tls-certs\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.451636 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.451612    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qvj\" (UniqueName: \"kubernetes.io/projected/77bc4a78-b956-45a2-b117-fe4efac876a7-kube-api-access-79qvj\") pod \"auth-test-model-8a4eb7-kserve-784599bff9-swh4g\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.568381 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.568289    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:32.838663 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:32.838632    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"]
Sep 11 19:57:32.840290 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:57:32.840264    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77bc4a78_b956_45a2_b117_fe4efac876a7.slice/crio-4b1e2088957528a1b88158dc535729ea6fd1a39983082f75a7641a2b8d887572 WatchSource:0}: Error finding container 4b1e2088957528a1b88158dc535729ea6fd1a39983082f75a7641a2b8d887572: Status 404 returned error can't find the container with id 4b1e2088957528a1b88158dc535729ea6fd1a39983082f75a7641a2b8d887572
Sep 11 19:57:33.687918 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:33.687885    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g" event={"ID":"77bc4a78-b956-45a2-b117-fe4efac876a7","Type":"ContainerStarted","Data":"3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1"}
Sep 11 19:57:33.688391 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:33.687926    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g" event={"ID":"77bc4a78-b956-45a2-b117-fe4efac876a7","Type":"ContainerStarted","Data":"4b1e2088957528a1b88158dc535729ea6fd1a39983082f75a7641a2b8d887572"}
Sep 11 19:57:33.707213 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:33.707154    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g" podStartSLOduration=1.7071391980000001 podStartE2EDuration="1.707139198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:57:33.704603285 +0000 UTC m=+1690.380540193" watchObservedRunningTime="2026-09-11 19:57:33.707139198 +0000 UTC m=+1690.383076121"
Sep 11 19:57:37.064152 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:37.064087    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:37.064522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:37.064190    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:37.077031 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:37.077005    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:37.722744 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:37.722716    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 19:57:41.036492 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.036457    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"]
Sep 11 19:57:41.036955 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.036664    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr" podUID="b9550029-c9ff-40b9-b12f-40260c9de2e6" containerName="payload-processing" containerID="cri-o://cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0" gracePeriod=30
Sep 11 19:57:41.207403 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.207359    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"]
Sep 11 19:57:41.209754 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.209733    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-105b25aa-mqwgc"]
Sep 11 19:57:41.289957 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.289889    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:57:41.429239 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.429202    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9550029-c9ff-40b9-b12f-40260c9de2e6-plugins-config-volume\") pod \"b9550029-c9ff-40b9-b12f-40260c9de2e6\" (UID: \"b9550029-c9ff-40b9-b12f-40260c9de2e6\") "
Sep 11 19:57:41.429389 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.429306    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lsj5\" (UniqueName: \"kubernetes.io/projected/b9550029-c9ff-40b9-b12f-40260c9de2e6-kube-api-access-5lsj5\") pod \"b9550029-c9ff-40b9-b12f-40260c9de2e6\" (UID: \"b9550029-c9ff-40b9-b12f-40260c9de2e6\") "
Sep 11 19:57:41.429601 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.429577    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9550029-c9ff-40b9-b12f-40260c9de2e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9550029-c9ff-40b9-b12f-40260c9de2e6" (UID: "b9550029-c9ff-40b9-b12f-40260c9de2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:57:41.431404 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.431380    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9550029-c9ff-40b9-b12f-40260c9de2e6-kube-api-access-5lsj5" (OuterVolumeSpecName: "kube-api-access-5lsj5") pod "b9550029-c9ff-40b9-b12f-40260c9de2e6" (UID: "b9550029-c9ff-40b9-b12f-40260c9de2e6"). InnerVolumeSpecName "kube-api-access-5lsj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:57:41.530144 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.530078    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9550029-c9ff-40b9-b12f-40260c9de2e6-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:41.530144 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.530140    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lsj5\" (UniqueName: \"kubernetes.io/projected/b9550029-c9ff-40b9-b12f-40260c9de2e6-kube-api-access-5lsj5\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:57:41.733567 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.733535    2571 generic.go:356] "Generic (PLEG): container finished" podID="b9550029-c9ff-40b9-b12f-40260c9de2e6" containerID="cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0" exitCode=0
Sep 11 19:57:41.733757 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.733586    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"
Sep 11 19:57:41.733757 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.733627    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr" event={"ID":"b9550029-c9ff-40b9-b12f-40260c9de2e6","Type":"ContainerDied","Data":"cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0"}
Sep 11 19:57:41.733757 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.733705    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr" event={"ID":"b9550029-c9ff-40b9-b12f-40260c9de2e6","Type":"ContainerDied","Data":"de7894163cac3d63c038b90084b63e7ae3c56dd2e3a41fdd1e2020f43d4a9974"}
Sep 11 19:57:41.733757 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.733726    2571 scope.go:117] "RemoveContainer" containerID="cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0"
Sep 11 19:57:41.745523 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.745496    2571 scope.go:117] "RemoveContainer" containerID="cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0"
Sep 11 19:57:41.746056 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:57:41.746032    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0\": container with ID starting with cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0 not found: ID does not exist" containerID="cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0"
Sep 11 19:57:41.746173 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.746064    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0"} err="failed to get container status \"cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0\": rpc error: code = NotFound desc = could not find container \"cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0\": container with ID starting with cb0c424394bd8e82fdb2a264a44b9adc8a74ce38cb89e6069831e68ac44bf8b0 not found: ID does not exist"
Sep 11 19:57:41.757084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.757056    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"]
Sep 11 19:57:41.762343 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.762308    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-105b25aa-7d7cc5fbcc-khmpr"]
Sep 11 19:57:41.874162 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.874096    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69051be5-856f-44bb-a5ab-86f5340aadaf" path="/var/lib/kubelet/pods/69051be5-856f-44bb-a5ab-86f5340aadaf/volumes"
Sep 11 19:57:41.874538 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:41.874524    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9550029-c9ff-40b9-b12f-40260c9de2e6" path="/var/lib/kubelet/pods/b9550029-c9ff-40b9-b12f-40260c9de2e6/volumes"
Sep 11 19:57:42.569065 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:42.569027    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:42.569580 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:42.569086    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:42.581511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:42.581482    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:42.751084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:42.751051    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 19:57:49.241140 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:57:49.241043    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"]
Sep 11 19:58:04.154267 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.154231    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:58:04.260292 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.260248    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm" podUID="aba22e26-b6ef-43b8-9b88-a21e811bdf25" containerName="main" containerID="cri-o://4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24" gracePeriod=45
Sep 11 19:58:04.500854 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.500826    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:58:04.538514 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.538461    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/aba22e26-b6ef-43b8-9b88-a21e811bdf25-kube-api-access-w8h4h\") pod \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") "
Sep 11 19:58:04.538514 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.538508    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-home\") pod \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") "
Sep 11 19:58:04.538745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.538574    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tmp-dir\") pod \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") "
Sep 11 19:58:04.538745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.538609    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-dshm\") pod \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") "
Sep 11 19:58:04.538745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.538696    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-model-cache\") pod \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") "
Sep 11 19:58:04.538745 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.538730    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tls-certs\") pod \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\" (UID: \"aba22e26-b6ef-43b8-9b88-a21e811bdf25\") "
Sep 11 19:58:04.538931 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.538815    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-home" (OuterVolumeSpecName: "home") pod "aba22e26-b6ef-43b8-9b88-a21e811bdf25" (UID: "aba22e26-b6ef-43b8-9b88-a21e811bdf25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:04.539292 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.539065    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:04.539518 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.539472    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aba22e26-b6ef-43b8-9b88-a21e811bdf25" (UID: "aba22e26-b6ef-43b8-9b88-a21e811bdf25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:04.539657 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.539632    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-model-cache" (OuterVolumeSpecName: "model-cache") pod "aba22e26-b6ef-43b8-9b88-a21e811bdf25" (UID: "aba22e26-b6ef-43b8-9b88-a21e811bdf25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:04.541266 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.541227    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aba22e26-b6ef-43b8-9b88-a21e811bdf25" (UID: "aba22e26-b6ef-43b8-9b88-a21e811bdf25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:58:04.541368 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.541327    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba22e26-b6ef-43b8-9b88-a21e811bdf25-kube-api-access-w8h4h" (OuterVolumeSpecName: "kube-api-access-w8h4h") pod "aba22e26-b6ef-43b8-9b88-a21e811bdf25" (UID: "aba22e26-b6ef-43b8-9b88-a21e811bdf25"). InnerVolumeSpecName "kube-api-access-w8h4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:58:04.541473 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.541442    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-dshm" (OuterVolumeSpecName: "dshm") pod "aba22e26-b6ef-43b8-9b88-a21e811bdf25" (UID: "aba22e26-b6ef-43b8-9b88-a21e811bdf25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:04.640087 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.640053    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:04.640087 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.640081    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:04.640087 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.640091    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba22e26-b6ef-43b8-9b88-a21e811bdf25-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:04.640380 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.640135    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba22e26-b6ef-43b8-9b88-a21e811bdf25-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:04.640380 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.640146    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8h4h\" (UniqueName: \"kubernetes.io/projected/aba22e26-b6ef-43b8-9b88-a21e811bdf25-kube-api-access-w8h4h\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:04.865749 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.865661    2571 generic.go:356] "Generic (PLEG): container finished" podID="aba22e26-b6ef-43b8-9b88-a21e811bdf25" containerID="4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24" exitCode=0
Sep 11 19:58:04.865749 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.865702    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"
Sep 11 19:58:04.866038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.865743    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm" event={"ID":"aba22e26-b6ef-43b8-9b88-a21e811bdf25","Type":"ContainerDied","Data":"4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24"}
Sep 11 19:58:04.866038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.865795    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm" event={"ID":"aba22e26-b6ef-43b8-9b88-a21e811bdf25","Type":"ContainerDied","Data":"65be374620688b3bec1cc0cc0cd3e3931821a3fc5bbc8be44267d4f1b94fc35d"}
Sep 11 19:58:04.866038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.865814    2571 scope.go:117] "RemoveContainer" containerID="4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24"
Sep 11 19:58:04.875349 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.875329    2571 scope.go:117] "RemoveContainer" containerID="4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24"
Sep 11 19:58:04.875608 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:58:04.875589    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24\": container with ID starting with 4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24 not found: ID does not exist" containerID="4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24"
Sep 11 19:58:04.875679 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.875617    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24"} err="failed to get container status \"4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24\": rpc error: code = NotFound desc = could not find container \"4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24\": container with ID starting with 4063298615d022e128836223b4bf33c5414d4a46651f0204339a4a8cc9f54e24 not found: ID does not exist"
Sep 11 19:58:04.888052 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.888029    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"]
Sep 11 19:58:04.891633 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:04.891612    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-105b25aa/e2e-lifecycle-model-105b25aa-kserve-65d467599f-zb2dm"]
Sep 11 19:58:05.873614 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:05.873579    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba22e26-b6ef-43b8-9b88-a21e811bdf25" path="/var/lib/kubelet/pods/aba22e26-b6ef-43b8-9b88-a21e811bdf25/volumes"
Sep 11 19:58:31.548256 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:31.548222    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:58:32.443150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.441212    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"]
Sep 11 19:58:32.443150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.442000    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba22e26-b6ef-43b8-9b88-a21e811bdf25" containerName="main"
Sep 11 19:58:32.443150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.442022    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba22e26-b6ef-43b8-9b88-a21e811bdf25" containerName="main"
Sep 11 19:58:32.443150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.442081    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9550029-c9ff-40b9-b12f-40260c9de2e6" containerName="payload-processing"
Sep 11 19:58:32.443150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.442092    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9550029-c9ff-40b9-b12f-40260c9de2e6" containerName="payload-processing"
Sep 11 19:58:32.443150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.442276    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9550029-c9ff-40b9-b12f-40260c9de2e6" containerName="payload-processing"
Sep 11 19:58:32.443150 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.442290    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba22e26-b6ef-43b8-9b88-a21e811bdf25" containerName="main"
Sep 11 19:58:32.446737 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.446706    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.450083 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.450058    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b032e\"/\"embedding-w0-4b032e-kserve-self-signed-certs\""
Sep 11 19:58:32.456034 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.456012    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"]
Sep 11 19:58:32.605046 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.605011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4699764c-395c-4b7b-b7ae-ff752ae93302-tls-certs\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.605436 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.605057    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-tmp-dir\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.605436 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.605076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-home\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.605436 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.605178    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbww5\" (UniqueName: \"kubernetes.io/projected/4699764c-395c-4b7b-b7ae-ff752ae93302-kube-api-access-nbww5\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.605436 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.605207    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-dshm\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.605436 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.605228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-model-cache\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.706657 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.706580    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4699764c-395c-4b7b-b7ae-ff752ae93302-tls-certs\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.706657 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.706621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-tmp-dir\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.706657 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.706637    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-home\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.706880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.706677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbww5\" (UniqueName: \"kubernetes.io/projected/4699764c-395c-4b7b-b7ae-ff752ae93302-kube-api-access-nbww5\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.706880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.706697    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-dshm\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.706880 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.706716    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-model-cache\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.707084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.707059    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-tmp-dir\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.707184 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.707089    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-home\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.707184 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.707128    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-model-cache\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.709021 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.709000    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-dshm\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.709260 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.709240    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4699764c-395c-4b7b-b7ae-ff752ae93302-tls-certs\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.714987 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.714966    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbww5\" (UniqueName: \"kubernetes.io/projected/4699764c-395c-4b7b-b7ae-ff752ae93302-kube-api-access-nbww5\") pod \"embedding-w0-4b032e-kserve-5b7c459d7d-924f5\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") " pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:32.757430 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:32.757402    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:33.030525 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:33.030498    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"]
Sep 11 19:58:33.031474 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:58:33.031449    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4699764c_395c_4b7b_b7ae_ff752ae93302.slice/crio-b20a956c53a7985f075ca558b2f8dd7ca285efb2b3e93adb36500b8d6adfd733 WatchSource:0}: Error finding container b20a956c53a7985f075ca558b2f8dd7ca285efb2b3e93adb36500b8d6adfd733: Status 404 returned error can't find the container with id b20a956c53a7985f075ca558b2f8dd7ca285efb2b3e93adb36500b8d6adfd733
Sep 11 19:58:34.021215 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:34.021178    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5" event={"ID":"4699764c-395c-4b7b-b7ae-ff752ae93302","Type":"ContainerStarted","Data":"f9dd2106d178abefc5cf8a2df9278831266a2ba9daa60c8087f6285142e349aa"}
Sep 11 19:58:34.021215 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:34.021220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5" event={"ID":"4699764c-395c-4b7b-b7ae-ff752ae93302","Type":"ContainerStarted","Data":"b20a956c53a7985f075ca558b2f8dd7ca285efb2b3e93adb36500b8d6adfd733"}
Sep 11 19:58:34.039143 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:34.039077    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5" podStartSLOduration=2.039061316 podStartE2EDuration="2.039061316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:58:34.036597734 +0000 UTC m=+1750.712534655" watchObservedRunningTime="2026-09-11 19:58:34.039061316 +0000 UTC m=+1750.714998266"
Sep 11 19:58:42.758192 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:42.758158    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:42.758681 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:42.758228    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:42.771047 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:42.771027    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:43.080380 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:43.080300    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 19:58:47.288727 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:47.288690    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"]
Sep 11 19:58:47.289229 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:47.288944    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy" containerID="cri-o://6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d" gracePeriod=30
Sep 11 19:58:52.531959 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.531931    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:58:52.567255 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567194    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-certs\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567255 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567239    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvrcg\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-kube-api-access-gvrcg\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567270    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istiod-ca-cert\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567292    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-credential-socket\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567332    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-podinfo\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567381    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-token\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567454 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567411    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-data\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567712 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567490    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:52.567712 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567605    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:52.567712 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567612    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-socket\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567712 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567666    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-envoy\") pod \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\" (UID: \"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f\") "
Sep 11 19:58:52.567712 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567674    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:58:52.567948 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.567887    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-data" (OuterVolumeSpecName: "istio-data") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:52.568144 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.568121    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:52.568272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.568147    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:52.568272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.568162    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:52.568272 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.568176    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:52.568522 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.568396    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:52.569785 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.569760    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-kube-api-access-gvrcg" (OuterVolumeSpecName: "kube-api-access-gvrcg") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "kube-api-access-gvrcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:58:52.569912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.569777    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-token" (OuterVolumeSpecName: "istio-token") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:58:52.569912 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.569850    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:58:52.570148 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.569983    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" (UID: "c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:58:52.669465 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.669443    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:52.669465 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.669464    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:52.669595 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.669472    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:52.669595 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.669481    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvrcg\" (UniqueName: \"kubernetes.io/projected/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-kube-api-access-gvrcg\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:52.669595 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:52.669491    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 19:58:53.128014 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.127970    2571 generic.go:356] "Generic (PLEG): container finished" podID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerID="6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d" exitCode=0
Sep 11 19:58:53.128222 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.128018    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" event={"ID":"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f","Type":"ContainerDied","Data":"6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d"}
Sep 11 19:58:53.128222 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.128110    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw" event={"ID":"c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f","Type":"ContainerDied","Data":"2cc330973c5a39420d9c7ee9f9a129acd85452ed82b9b23de20b5c74690bbc0b"}
Sep 11 19:58:53.128222 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.128130    2571 scope.go:117] "RemoveContainer" containerID="6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d"
Sep 11 19:58:53.128409 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.128332    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"
Sep 11 19:58:53.138380 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.138362    2571 scope.go:117] "RemoveContainer" containerID="f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14"
Sep 11 19:58:53.148489 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.148474    2571 scope.go:117] "RemoveContainer" containerID="6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d"
Sep 11 19:58:53.148761 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:58:53.148745    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d\": container with ID starting with 6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d not found: ID does not exist" containerID="6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d"
Sep 11 19:58:53.148807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.148769    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d"} err="failed to get container status \"6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d\": rpc error: code = NotFound desc = could not find container \"6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d\": container with ID starting with 6a2272d4f95c6db9fa13c662d4f31a2d57b98e6d8adbc3ed326f181d2eb7626d not found: ID does not exist"
Sep 11 19:58:53.148807 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.148785    2571 scope.go:117] "RemoveContainer" containerID="f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14"
Sep 11 19:58:53.149033 ip-10-0-136-27 kubenswrapper[2571]: E0911 19:58:53.149017    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14\": container with ID starting with f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14 not found: ID does not exist" containerID="f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14"
Sep 11 19:58:53.149084 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.149036    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14"} err="failed to get container status \"f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14\": rpc error: code = NotFound desc = could not find container \"f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14\": container with ID starting with f2fa95d4c724f407e2148ce9f372e795eeb1b4c9abe12eeb2984c03afccd1c14 not found: ID does not exist"
Sep 11 19:58:53.152197 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.152179    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"]
Sep 11 19:58:53.156466 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.156445    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-105b25aa-openshift-default-74fc4f5968-kfblw"]
Sep 11 19:58:53.873456 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:58:53.873424    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" path="/var/lib/kubelet/pods/c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f/volumes"
Sep 11 19:59:37.948152 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:37.948118    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 19:59:42.536729 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.536699    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"]
Sep 11 19:59:42.537111 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.537009    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy"
Sep 11 19:59:42.537111 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.537020    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy"
Sep 11 19:59:42.537111 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.537039    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy"
Sep 11 19:59:42.537111 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.537045    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy"
Sep 11 19:59:42.537269 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.537116    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy"
Sep 11 19:59:42.537269 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.537127    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e0ac38-4c51-4bb9-b40d-4d30c367ce4f" containerName="istio-proxy"
Sep 11 19:59:42.540147 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.540130    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.542741 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.542716    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 19:59:42.542845 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.542775    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e7cdd6-openshift-default-dockercfg-7hcf2\""
Sep 11 19:59:42.561447 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.561425    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"]
Sep 11 19:59:42.587205 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587183    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-certs\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.587326 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-podinfo\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.587326 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587285    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-token\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.587442 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587339    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-envoy\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.587442 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-data\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.587442 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587431    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-socket\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.587555 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6wz\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-kube-api-access-pr6wz\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.587555 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd493de-1d79-427c-aa36-b2f2725a4a44-istiod-ca-cert\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.587555 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.587508    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-credential-socket\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.687913 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.687885    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6wz\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-kube-api-access-pr6wz\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.687920    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd493de-1d79-427c-aa36-b2f2725a4a44-istiod-ca-cert\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.687949    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-credential-socket\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.687977    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-certs\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688038 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688029    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-podinfo\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688311 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688069    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-token\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688311 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-envoy\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688311 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-data\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688311 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-socket\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688532 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688465    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-certs\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688532 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688521    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-credential-socket\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688634 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688582    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-socket\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688677 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688642    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-data\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.688712 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.688685    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd493de-1d79-427c-aa36-b2f2725a4a44-istiod-ca-cert\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.690849 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.690829    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-envoy\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.690969 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.690951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-podinfo\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.697360 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.697340    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-token\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.697955 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.697935    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6wz\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-kube-api-access-pr6wz\") pod \"e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") " pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:42.850541 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:42.850477    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:43.130526 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:43.130495    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"]
Sep 11 19:59:43.132040 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:59:43.132016    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd493de_1d79_427c_aa36_b2f2725a4a44.slice/crio-175d21989938a83a7be711709108d0e6a0104f2a1d36451f36586a4254acd909 WatchSource:0}: Error finding container 175d21989938a83a7be711709108d0e6a0104f2a1d36451f36586a4254acd909: Status 404 returned error can't find the container with id 175d21989938a83a7be711709108d0e6a0104f2a1d36451f36586a4254acd909
Sep 11 19:59:43.134724 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:43.134693    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:59:43.134830 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:43.134769    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:59:43.134830 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:43.134815    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 19:59:43.399918 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:43.399820    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" event={"ID":"ddd493de-1d79-427c-aa36-b2f2725a4a44","Type":"ContainerStarted","Data":"aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d"}
Sep 11 19:59:43.399918 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:43.399885    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" event={"ID":"ddd493de-1d79-427c-aa36-b2f2725a4a44","Type":"ContainerStarted","Data":"175d21989938a83a7be711709108d0e6a0104f2a1d36451f36586a4254acd909"}
Sep 11 19:59:43.422905 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:43.422852    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" podStartSLOduration=1.422836095 podStartE2EDuration="1.422836095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:59:43.419849651 +0000 UTC m=+1820.095786559" watchObservedRunningTime="2026-09-11 19:59:43.422836095 +0000 UTC m=+1820.098773072"
Sep 11 19:59:43.851519 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:43.851439    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:44.856185 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:44.856155    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:45.409922 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:45.409889    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:45.410897 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:45.410872    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 19:59:50.352511 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.352429    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"]
Sep 11 19:59:50.355775 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.355758    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:50.358471 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.358444    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e7cdd6\""
Sep 11 19:59:50.358609 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.358553    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e7cdd6-dockercfg-zhbt4\""
Sep 11 19:59:50.366849 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.366826    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"]
Sep 11 19:59:50.454869 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.454836    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddh4w\" (UniqueName: \"kubernetes.io/projected/d157afbf-dbea-41a9-8035-8f6140d02f9a-kube-api-access-ddh4w\") pod \"payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg\" (UID: \"d157afbf-dbea-41a9-8035-8f6140d02f9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:50.455023 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.454891    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d157afbf-dbea-41a9-8035-8f6140d02f9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg\" (UID: \"d157afbf-dbea-41a9-8035-8f6140d02f9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:50.556010 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.555979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d157afbf-dbea-41a9-8035-8f6140d02f9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg\" (UID: \"d157afbf-dbea-41a9-8035-8f6140d02f9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:50.556178 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.556077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddh4w\" (UniqueName: \"kubernetes.io/projected/d157afbf-dbea-41a9-8035-8f6140d02f9a-kube-api-access-ddh4w\") pod \"payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg\" (UID: \"d157afbf-dbea-41a9-8035-8f6140d02f9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:50.556660 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.556640    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d157afbf-dbea-41a9-8035-8f6140d02f9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg\" (UID: \"d157afbf-dbea-41a9-8035-8f6140d02f9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:50.569705 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.569682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddh4w\" (UniqueName: \"kubernetes.io/projected/d157afbf-dbea-41a9-8035-8f6140d02f9a-kube-api-access-ddh4w\") pod \"payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg\" (UID: \"d157afbf-dbea-41a9-8035-8f6140d02f9a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:50.667199 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.667171    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:50.936883 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:50.936861    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"]
Sep 11 19:59:50.938434 ip-10-0-136-27 kubenswrapper[2571]: W0911 19:59:50.938408    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd157afbf_dbea_41a9_8035_8f6140d02f9a.slice/crio-9db7d32c352650e328f265088b2d2f49dfb0e43507fc000ccdf130b1a59a9eb2 WatchSource:0}: Error finding container 9db7d32c352650e328f265088b2d2f49dfb0e43507fc000ccdf130b1a59a9eb2: Status 404 returned error can't find the container with id 9db7d32c352650e328f265088b2d2f49dfb0e43507fc000ccdf130b1a59a9eb2
Sep 11 19:59:51.443959 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:51.443924    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg" event={"ID":"d157afbf-dbea-41a9-8035-8f6140d02f9a","Type":"ContainerStarted","Data":"56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf"}
Sep 11 19:59:51.444370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:51.444007    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg" event={"ID":"d157afbf-dbea-41a9-8035-8f6140d02f9a","Type":"ContainerStarted","Data":"9db7d32c352650e328f265088b2d2f49dfb0e43507fc000ccdf130b1a59a9eb2"}
Sep 11 19:59:51.444370 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:51.444084    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 19:59:51.461851 ip-10-0-136-27 kubenswrapper[2571]: I0911 19:59:51.461804    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg" podStartSLOduration=1.46179087 podStartE2EDuration="1.46179087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:59:51.459425832 +0000 UTC m=+1828.135362738" watchObservedRunningTime="2026-09-11 19:59:51.46179087 +0000 UTC m=+1828.137727777"
Sep 11 20:00:00.139582 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.139545    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"]
Sep 11 20:00:00.143161 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.143131    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"
Sep 11 20:00:00.143739 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.143708    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"]
Sep 11 20:00:00.146235 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.146213    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:00:00.147299 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.147283    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"]
Sep 11 20:00:00.147395 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.147379    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"
Sep 11 20:00:00.150405 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.150377    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"
Sep 11 20:00:00.154923 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.154903    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"]
Sep 11 20:00:00.155156 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.155137    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"
Sep 11 20:00:00.163329 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.163287    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"]
Sep 11 20:00:00.165426 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.165409    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"
Sep 11 20:00:00.166769 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.165980    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"]
Sep 11 20:00:00.173978 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.173955    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"]
Sep 11 20:00:00.178765 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.178743    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"
Sep 11 20:00:00.192185 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.192156    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"]
Sep 11 20:00:00.207993 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.207965    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"
Sep 11 20:00:00.215685 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.215663    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"
Sep 11 20:00:00.450054 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.450031    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"]
Sep 11 20:00:00.451009 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:00:00.450987    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce813680_4d39_43ac_90b7_335b7591ea84.slice/crio-29b5fd1f70127075128a27690bfd9815a9395890aa17bfba8e7eaf1c89345ccc WatchSource:0}: Error finding container 29b5fd1f70127075128a27690bfd9815a9395890aa17bfba8e7eaf1c89345ccc: Status 404 returned error can't find the container with id 29b5fd1f70127075128a27690bfd9815a9395890aa17bfba8e7eaf1c89345ccc
Sep 11 20:00:00.459464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.459438    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"]
Sep 11 20:00:00.460375 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:00:00.460352    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74273c4b_9aeb_4f0e_a7d6_3f0fc71176d2.slice/crio-71e83fb5a20615524ec3a828c61f83e6f2a96462df8389320c4180a930f173ae WatchSource:0}: Error finding container 71e83fb5a20615524ec3a828c61f83e6f2a96462df8389320c4180a930f173ae: Status 404 returned error can't find the container with id 71e83fb5a20615524ec3a828c61f83e6f2a96462df8389320c4180a930f173ae
Sep 11 20:00:00.494144 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.494117    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs" event={"ID":"74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2","Type":"ContainerStarted","Data":"71e83fb5a20615524ec3a828c61f83e6f2a96462df8389320c4180a930f173ae"}
Sep 11 20:00:00.495584 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.495563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m" event={"ID":"ce813680-4d39-43ac-90b7-335b7591ea84","Type":"ContainerStarted","Data":"29b5fd1f70127075128a27690bfd9815a9395890aa17bfba8e7eaf1c89345ccc"}
Sep 11 20:00:00.518090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.518039    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"]
Sep 11 20:00:00.519797 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:00:00.519771    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861b2cb7_ec6a_4e46_897a_a54ad8ccfaec.slice/crio-08a28ba70c89fbcdcbaad6ba59f46df70aff7a50963a62da90c9b0a4fd48273e WatchSource:0}: Error finding container 08a28ba70c89fbcdcbaad6ba59f46df70aff7a50963a62da90c9b0a4fd48273e: Status 404 returned error can't find the container with id 08a28ba70c89fbcdcbaad6ba59f46df70aff7a50963a62da90c9b0a4fd48273e
Sep 11 20:00:00.521330 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:00.521311    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"]
Sep 11 20:00:00.522293 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:00:00.522270    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acb00c7_b7e0_432d_9fca_110732e48033.slice/crio-336f35ea1fe03a618bc5b23a50dc4959f3648fee5c0e24a8a9a6e6f4efa65b57 WatchSource:0}: Error finding container 336f35ea1fe03a618bc5b23a50dc4959f3648fee5c0e24a8a9a6e6f4efa65b57: Status 404 returned error can't find the container with id 336f35ea1fe03a618bc5b23a50dc4959f3648fee5c0e24a8a9a6e6f4efa65b57
Sep 11 20:00:01.503344 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.503311    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk" event={"ID":"6acb00c7-b7e0-432d-9fca-110732e48033","Type":"ContainerStarted","Data":"47a6fbd0c6fc0889d23f2cb9f3b08369cac98b50e58dcf0585252e0bae005d7c"}
Sep 11 20:00:01.503758 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.503400    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk" event={"ID":"6acb00c7-b7e0-432d-9fca-110732e48033","Type":"ContainerStarted","Data":"336f35ea1fe03a618bc5b23a50dc4959f3648fee5c0e24a8a9a6e6f4efa65b57"}
Sep 11 20:00:01.504859 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.504829    2571 generic.go:356] "Generic (PLEG): container finished" podID="74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2" containerID="f59f8ca35187edd26a61d607c1a67306e20af7d6f1b12e945c5f6af88aef74d9" exitCode=0
Sep 11 20:00:01.504975 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.504924    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs" event={"ID":"74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2","Type":"ContainerDied","Data":"f59f8ca35187edd26a61d607c1a67306e20af7d6f1b12e945c5f6af88aef74d9"}
Sep 11 20:00:01.506423 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.506401    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq" event={"ID":"861b2cb7-ec6a-4e46-897a-a54ad8ccfaec","Type":"ContainerStarted","Data":"bb29c1c61dcc2f2e6a013f802258a5792b7fbf0f5fe9fcdecc6ce4271e1809c6"}
Sep 11 20:00:01.506767 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.506439    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq" event={"ID":"861b2cb7-ec6a-4e46-897a-a54ad8ccfaec","Type":"ContainerStarted","Data":"08a28ba70c89fbcdcbaad6ba59f46df70aff7a50963a62da90c9b0a4fd48273e"}
Sep 11 20:00:01.508216 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.508192    2571 generic.go:356] "Generic (PLEG): container finished" podID="ce813680-4d39-43ac-90b7-335b7591ea84" containerID="54a4176e686da8c5df46bada8c1082c40541c85b0d12ca4e197d6df4e481b9ec" exitCode=0
Sep 11 20:00:01.508305 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.508234    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m" event={"ID":"ce813680-4d39-43ac-90b7-335b7591ea84","Type":"ContainerDied","Data":"54a4176e686da8c5df46bada8c1082c40541c85b0d12ca4e197d6df4e481b9ec"}
Sep 11 20:00:01.519414 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.519361    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk" podStartSLOduration=1.519347448 podStartE2EDuration="1.519347448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:01.517839689 +0000 UTC m=+1838.193776598" watchObservedRunningTime="2026-09-11 20:00:01.519347448 +0000 UTC m=+1838.195284393"
Sep 11 20:00:01.559028 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.558983    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq" podStartSLOduration=1.5589697230000001 podStartE2EDuration="1.558969723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:01.558158644 +0000 UTC m=+1838.234095556" watchObservedRunningTime="2026-09-11 20:00:01.558969723 +0000 UTC m=+1838.234906630"
Sep 11 20:00:01.610253 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.610188    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"]
Sep 11 20:00:01.614319 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.614296    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"
Sep 11 20:00:01.620089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.617158    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:00:01.624543 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.624520    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"]
Sep 11 20:00:01.758329 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.758295    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230ef834-433d-4f8e-ad83-3d335e1742cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk\" (UID: \"230ef834-433d-4f8e-ad83-3d335e1742cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"
Sep 11 20:00:01.858934 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.858903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230ef834-433d-4f8e-ad83-3d335e1742cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk\" (UID: \"230ef834-433d-4f8e-ad83-3d335e1742cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"
Sep 11 20:00:01.859533 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:01.859516    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230ef834-433d-4f8e-ad83-3d335e1742cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk\" (UID: \"230ef834-433d-4f8e-ad83-3d335e1742cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"
Sep 11 20:00:02.032014 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.031975    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"
Sep 11 20:00:02.302990 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.302961    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"]
Sep 11 20:00:02.304299 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:00:02.304274    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230ef834_433d_4f8e_ad83_3d335e1742cf.slice/crio-1cb1b53582d05f667300d86606f3c548739ad3bff191b44a03294a13b0a0b6d1 WatchSource:0}: Error finding container 1cb1b53582d05f667300d86606f3c548739ad3bff191b44a03294a13b0a0b6d1: Status 404 returned error can't find the container with id 1cb1b53582d05f667300d86606f3c548739ad3bff191b44a03294a13b0a0b6d1
Sep 11 20:00:02.452805 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.452771    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 20:00:02.516328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.516298    2571 generic.go:356] "Generic (PLEG): container finished" podID="6acb00c7-b7e0-432d-9fca-110732e48033" containerID="47a6fbd0c6fc0889d23f2cb9f3b08369cac98b50e58dcf0585252e0bae005d7c" exitCode=0
Sep 11 20:00:02.516669 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.516388    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk" event={"ID":"6acb00c7-b7e0-432d-9fca-110732e48033","Type":"ContainerDied","Data":"47a6fbd0c6fc0889d23f2cb9f3b08369cac98b50e58dcf0585252e0bae005d7c"}
Sep 11 20:00:02.517944 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.517921    2571 generic.go:356] "Generic (PLEG): container finished" podID="861b2cb7-ec6a-4e46-897a-a54ad8ccfaec" containerID="bb29c1c61dcc2f2e6a013f802258a5792b7fbf0f5fe9fcdecc6ce4271e1809c6" exitCode=0
Sep 11 20:00:02.518053 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.517997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq" event={"ID":"861b2cb7-ec6a-4e46-897a-a54ad8ccfaec","Type":"ContainerDied","Data":"bb29c1c61dcc2f2e6a013f802258a5792b7fbf0f5fe9fcdecc6ce4271e1809c6"}
Sep 11 20:00:02.519461 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.519442    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk" event={"ID":"230ef834-433d-4f8e-ad83-3d335e1742cf","Type":"ContainerStarted","Data":"1cb1b53582d05f667300d86606f3c548739ad3bff191b44a03294a13b0a0b6d1"}
Sep 11 20:00:02.663784 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.663760    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"
Sep 11 20:00:02.677221 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:02.677194    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"
Sep 11 20:00:03.526864 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.526836    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"
Sep 11 20:00:03.527290 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.526836    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs" event={"ID":"74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2","Type":"ContainerDied","Data":"71e83fb5a20615524ec3a828c61f83e6f2a96462df8389320c4180a930f173ae"}
Sep 11 20:00:03.527290 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.527008    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71e83fb5a20615524ec3a828c61f83e6f2a96462df8389320c4180a930f173ae"
Sep 11 20:00:03.528388 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.528361    2571 generic.go:356] "Generic (PLEG): container finished" podID="230ef834-433d-4f8e-ad83-3d335e1742cf" containerID="ec4e6a826cd8ca2e59c1f93fdf0862da1038c203987e428a566d85d754494793" exitCode=0
Sep 11 20:00:03.528511 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.528443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk" event={"ID":"230ef834-433d-4f8e-ad83-3d335e1742cf","Type":"ContainerDied","Data":"ec4e6a826cd8ca2e59c1f93fdf0862da1038c203987e428a566d85d754494793"}
Sep 11 20:00:03.529823 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.529801    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"
Sep 11 20:00:03.529963 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.529835    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m" event={"ID":"ce813680-4d39-43ac-90b7-335b7591ea84","Type":"ContainerDied","Data":"29b5fd1f70127075128a27690bfd9815a9395890aa17bfba8e7eaf1c89345ccc"}
Sep 11 20:00:03.529963 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.529871    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b5fd1f70127075128a27690bfd9815a9395890aa17bfba8e7eaf1c89345ccc"
Sep 11 20:00:03.664425 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.664401    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"
Sep 11 20:00:03.694881 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:03.694852    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"
Sep 11 20:00:04.536888 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.536848    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"
Sep 11 20:00:04.536888 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.536868    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk" event={"ID":"6acb00c7-b7e0-432d-9fca-110732e48033","Type":"ContainerDied","Data":"336f35ea1fe03a618bc5b23a50dc4959f3648fee5c0e24a8a9a6e6f4efa65b57"}
Sep 11 20:00:04.537410 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.536915    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336f35ea1fe03a618bc5b23a50dc4959f3648fee5c0e24a8a9a6e6f4efa65b57"
Sep 11 20:00:04.538239 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.538221    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"
Sep 11 20:00:04.538346 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.538260    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq" event={"ID":"861b2cb7-ec6a-4e46-897a-a54ad8ccfaec","Type":"ContainerDied","Data":"08a28ba70c89fbcdcbaad6ba59f46df70aff7a50963a62da90c9b0a4fd48273e"}
Sep 11 20:00:04.538346 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.538303    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08a28ba70c89fbcdcbaad6ba59f46df70aff7a50963a62da90c9b0a4fd48273e"
Sep 11 20:00:04.664859 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.664834    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"
Sep 11 20:00:04.788896 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.788808    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230ef834-433d-4f8e-ad83-3d335e1742cf-maas-api-service-ca\") pod \"230ef834-433d-4f8e-ad83-3d335e1742cf\" (UID: \"230ef834-433d-4f8e-ad83-3d335e1742cf\") "
Sep 11 20:00:04.789170 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.789149    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230ef834-433d-4f8e-ad83-3d335e1742cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "230ef834-433d-4f8e-ad83-3d335e1742cf" (UID: "230ef834-433d-4f8e-ad83-3d335e1742cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:00:04.890235 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:04.890203    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230ef834-433d-4f8e-ad83-3d335e1742cf-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:05.545719 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:05.545689    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"
Sep 11 20:00:05.546135 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:05.545730    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk" event={"ID":"230ef834-433d-4f8e-ad83-3d335e1742cf","Type":"ContainerDied","Data":"1cb1b53582d05f667300d86606f3c548739ad3bff191b44a03294a13b0a0b6d1"}
Sep 11 20:00:05.546135 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:05.545789    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cb1b53582d05f667300d86606f3c548739ad3bff191b44a03294a13b0a0b6d1"
Sep 11 20:00:06.482666 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:06.482629    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"]
Sep 11 20:00:06.500039 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:06.500012    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"]
Sep 11 20:00:06.514955 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:06.514926    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"]
Sep 11 20:00:06.529301 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:06.529276    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"]
Sep 11 20:00:15.273664 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.273624    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"]
Sep 11 20:00:15.276007 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.275984    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b032e-29819280-2h8rq"]
Sep 11 20:00:15.308738 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.308707    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"]
Sep 11 20:00:15.308941 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.308914    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8" podUID="296f502b-f979-4b32-a645-55cf4c8fd02a" containerName="payload-pre-processing" containerID="cri-o://950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875" gracePeriod=30
Sep 11 20:00:15.565835 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.565812    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 20:00:15.605177 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.605146    2571 generic.go:356] "Generic (PLEG): container finished" podID="296f502b-f979-4b32-a645-55cf4c8fd02a" containerID="950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875" exitCode=0
Sep 11 20:00:15.605177 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.605182    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"
Sep 11 20:00:15.605386 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.605220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8" event={"ID":"296f502b-f979-4b32-a645-55cf4c8fd02a","Type":"ContainerDied","Data":"950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875"}
Sep 11 20:00:15.605386 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.605273    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8" event={"ID":"296f502b-f979-4b32-a645-55cf4c8fd02a","Type":"ContainerDied","Data":"eba376de1206c1635e471f9b95f1aa663488e499e4d26751b2f8855b34bdab18"}
Sep 11 20:00:15.605386 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.605301    2571 scope.go:117] "RemoveContainer" containerID="950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875"
Sep 11 20:00:15.614469 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.614447    2571 scope.go:117] "RemoveContainer" containerID="950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875"
Sep 11 20:00:15.614747 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:00:15.614727    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875\": container with ID starting with 950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875 not found: ID does not exist" containerID="950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875"
Sep 11 20:00:15.614802 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.614756    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875"} err="failed to get container status \"950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875\": rpc error: code = NotFound desc = could not find container \"950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875\": container with ID starting with 950b26912e2df3c8525332b5d66b83a1d31c1e9a90e35b470cab822cfa603875 not found: ID does not exist"
Sep 11 20:00:15.638073 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.638045    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"]
Sep 11 20:00:15.644266 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.644242    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b032e-4mtqk"]
Sep 11 20:00:15.693688 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.693649    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtwvk\" (UniqueName: \"kubernetes.io/projected/296f502b-f979-4b32-a645-55cf4c8fd02a-kube-api-access-rtwvk\") pod \"296f502b-f979-4b32-a645-55cf4c8fd02a\" (UID: \"296f502b-f979-4b32-a645-55cf4c8fd02a\") "
Sep 11 20:00:15.693856 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.693767    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296f502b-f979-4b32-a645-55cf4c8fd02a-plugins-config-volume\") pod \"296f502b-f979-4b32-a645-55cf4c8fd02a\" (UID: \"296f502b-f979-4b32-a645-55cf4c8fd02a\") "
Sep 11 20:00:15.694192 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.694139    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296f502b-f979-4b32-a645-55cf4c8fd02a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "296f502b-f979-4b32-a645-55cf4c8fd02a" (UID: "296f502b-f979-4b32-a645-55cf4c8fd02a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:00:15.695745 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.695725    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296f502b-f979-4b32-a645-55cf4c8fd02a-kube-api-access-rtwvk" (OuterVolumeSpecName: "kube-api-access-rtwvk") pod "296f502b-f979-4b32-a645-55cf4c8fd02a" (UID: "296f502b-f979-4b32-a645-55cf4c8fd02a"). InnerVolumeSpecName "kube-api-access-rtwvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:00:15.795674 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.795591    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtwvk\" (UniqueName: \"kubernetes.io/projected/296f502b-f979-4b32-a645-55cf4c8fd02a-kube-api-access-rtwvk\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:15.795674 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.795631    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296f502b-f979-4b32-a645-55cf4c8fd02a-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:15.874552 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.874515    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230ef834-433d-4f8e-ad83-3d335e1742cf" path="/var/lib/kubelet/pods/230ef834-433d-4f8e-ad83-3d335e1742cf/volumes"
Sep 11 20:00:15.874832 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.874819    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="861b2cb7-ec6a-4e46-897a-a54ad8ccfaec" path="/var/lib/kubelet/pods/861b2cb7-ec6a-4e46-897a-a54ad8ccfaec/volumes"
Sep 11 20:00:15.922332 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.922306    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"]
Sep 11 20:00:15.926318 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:15.926295    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4b032e-7794845bc4-tn7x8"]
Sep 11 20:00:17.873993 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:17.873954    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="296f502b-f979-4b32-a645-55cf4c8fd02a" path="/var/lib/kubelet/pods/296f502b-f979-4b32-a645-55cf4c8fd02a/volumes"
Sep 11 20:00:19.909919 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.909875    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"]
Sep 11 20:00:19.910441 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910396    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296f502b-f979-4b32-a645-55cf4c8fd02a" containerName="payload-pre-processing"
Sep 11 20:00:19.910441 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910416    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="296f502b-f979-4b32-a645-55cf4c8fd02a" containerName="payload-pre-processing"
Sep 11 20:00:19.910441 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910429    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce813680-4d39-43ac-90b7-335b7591ea84" containerName="cleanup"
Sep 11 20:00:19.910441 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910438    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce813680-4d39-43ac-90b7-335b7591ea84" containerName="cleanup"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910457    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6acb00c7-b7e0-432d-9fca-110732e48033" containerName="cleanup"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910466    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6acb00c7-b7e0-432d-9fca-110732e48033" containerName="cleanup"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910479    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861b2cb7-ec6a-4e46-897a-a54ad8ccfaec" containerName="cleanup"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910488    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="861b2cb7-ec6a-4e46-897a-a54ad8ccfaec" containerName="cleanup"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910501    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230ef834-433d-4f8e-ad83-3d335e1742cf" containerName="revoke-keys"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910510    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="230ef834-433d-4f8e-ad83-3d335e1742cf" containerName="revoke-keys"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910521    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2" containerName="cleanup"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910530    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2" containerName="cleanup"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910653    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="230ef834-433d-4f8e-ad83-3d335e1742cf" containerName="revoke-keys"
Sep 11 20:00:19.910665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910664    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="861b2cb7-ec6a-4e46-897a-a54ad8ccfaec" containerName="cleanup"
Sep 11 20:00:19.911150 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910675    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6acb00c7-b7e0-432d-9fca-110732e48033" containerName="cleanup"
Sep 11 20:00:19.911150 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910685    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2" containerName="cleanup"
Sep 11 20:00:19.911150 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910698    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce813680-4d39-43ac-90b7-335b7591ea84" containerName="cleanup"
Sep 11 20:00:19.911150 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.910710    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="296f502b-f979-4b32-a645-55cf4c8fd02a" containerName="payload-pre-processing"
Sep 11 20:00:19.915269 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.915246    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:19.917980 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.917954    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b1a074-dockercfg-vf7x5\""
Sep 11 20:00:19.918096 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.917954    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b1a074\""
Sep 11 20:00:19.924613 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.924590    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"]
Sep 11 20:00:19.929573 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.929552    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"]
Sep 11 20:00:19.930958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.930933    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glz8h\" (UniqueName: \"kubernetes.io/projected/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-kube-api-access-glz8h\") pod \"payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s\" (UID: \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:19.931061 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.930983    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s\" (UID: \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:19.932915 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.932898    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:19.944246 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:19.944208    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"]
Sep 11 20:00:20.032301 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.032264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06186a8c-cae4-47cd-89c6-23b4f50a4a30-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw\" (UID: \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:20.032488 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.032326    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz8h\" (UniqueName: \"kubernetes.io/projected/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-kube-api-access-glz8h\") pod \"payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s\" (UID: \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:20.032488 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.032362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s\" (UID: \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:20.032488 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.032391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksj9x\" (UniqueName: \"kubernetes.io/projected/06186a8c-cae4-47cd-89c6-23b4f50a4a30-kube-api-access-ksj9x\") pod \"payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw\" (UID: \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:20.033124 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.033080    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s\" (UID: \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:20.042923 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.042898    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz8h\" (UniqueName: \"kubernetes.io/projected/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-kube-api-access-glz8h\") pod \"payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s\" (UID: \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:20.133382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.133345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06186a8c-cae4-47cd-89c6-23b4f50a4a30-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw\" (UID: \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:20.133561 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.133412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksj9x\" (UniqueName: \"kubernetes.io/projected/06186a8c-cae4-47cd-89c6-23b4f50a4a30-kube-api-access-ksj9x\") pod \"payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw\" (UID: \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:20.134015 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.133993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06186a8c-cae4-47cd-89c6-23b4f50a4a30-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw\" (UID: \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:20.140905 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.140881    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksj9x\" (UniqueName: \"kubernetes.io/projected/06186a8c-cae4-47cd-89c6-23b4f50a4a30-kube-api-access-ksj9x\") pod \"payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw\" (UID: \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:20.226947 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.226855    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:20.243747 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.243713    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:20.462528 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.462489    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:00:20.531622 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.531593    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"]
Sep 11 20:00:20.558026 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.557989    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"]
Sep 11 20:00:20.562407 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:00:20.562366    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06186a8c_cae4_47cd_89c6_23b4f50a4a30.slice/crio-4a10fde066819d52f9d90c79bbaa950e885de4589cdf393bbeb4b8550167115b WatchSource:0}: Error finding container 4a10fde066819d52f9d90c79bbaa950e885de4589cdf393bbeb4b8550167115b: Status 404 returned error can't find the container with id 4a10fde066819d52f9d90c79bbaa950e885de4589cdf393bbeb4b8550167115b
Sep 11 20:00:20.643488 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.643450    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw" event={"ID":"06186a8c-cae4-47cd-89c6-23b4f50a4a30","Type":"ContainerStarted","Data":"fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87"}
Sep 11 20:00:20.643679 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.643514    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw" event={"ID":"06186a8c-cae4-47cd-89c6-23b4f50a4a30","Type":"ContainerStarted","Data":"4a10fde066819d52f9d90c79bbaa950e885de4589cdf393bbeb4b8550167115b"}
Sep 11 20:00:20.643679 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.643652    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:20.645611 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.645580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s" event={"ID":"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd","Type":"ContainerStarted","Data":"77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072"}
Sep 11 20:00:20.645760 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.645629    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s" event={"ID":"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd","Type":"ContainerStarted","Data":"70f406a19cbcf8cf83d78606d4667a8a92fe05783ce510b94b8af73897678242"}
Sep 11 20:00:20.645760 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.645655    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:20.661370 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.661320    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw" podStartSLOduration=1.661304203 podStartE2EDuration="1.661304203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:20.659052373 +0000 UTC m=+1857.334989281" watchObservedRunningTime="2026-09-11 20:00:20.661304203 +0000 UTC m=+1857.337241145"
Sep 11 20:00:20.679984 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:20.679924    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s" podStartSLOduration=1.6799043930000002 podStartE2EDuration="1.679904393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:20.677140394 +0000 UTC m=+1857.353077301" watchObservedRunningTime="2026-09-11 20:00:20.679904393 +0000 UTC m=+1857.355841301"
Sep 11 20:00:21.501879 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.501813    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr" podUID="e558950d-0027-4a14-9600-df9dbb6e86dc" containerName="main" containerID="cri-o://97441a64d791f01aad43d88ef8a93e4b10f617ef0bcf4d71729bcafd55ac34a9" gracePeriod=45
Sep 11 20:00:21.520538 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.520509    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5" podUID="4699764c-395c-4b7b-b7ae-ff752ae93302" containerName="main" containerID="cri-o://f9dd2106d178abefc5cf8a2df9278831266a2ba9daa60c8087f6285142e349aa" gracePeriod=45
Sep 11 20:00:21.533762 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.533731    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6" podUID="22db71ec-b9b9-4de7-91b7-be46436e8270" containerName="main" containerID="cri-o://ab72ebc5262a2c4809912c1d8b8e14730e4d6088e0d70bdb25e01db1e677610e" gracePeriod=45
Sep 11 20:00:21.547787 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.547756    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm" podUID="162965eb-da36-444b-9305-b27bb7ac071c" containerName="main" containerID="cri-o://8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d" gracePeriod=45
Sep 11 20:00:21.653901 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.653705    2571 generic.go:356] "Generic (PLEG): container finished" podID="22db71ec-b9b9-4de7-91b7-be46436e8270" containerID="ab72ebc5262a2c4809912c1d8b8e14730e4d6088e0d70bdb25e01db1e677610e" exitCode=0
Sep 11 20:00:21.653901 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.653845    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6" event={"ID":"22db71ec-b9b9-4de7-91b7-be46436e8270","Type":"ContainerDied","Data":"ab72ebc5262a2c4809912c1d8b8e14730e4d6088e0d70bdb25e01db1e677610e"}
Sep 11 20:00:21.656159 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.656125    2571 generic.go:356] "Generic (PLEG): container finished" podID="4699764c-395c-4b7b-b7ae-ff752ae93302" containerID="f9dd2106d178abefc5cf8a2df9278831266a2ba9daa60c8087f6285142e349aa" exitCode=0
Sep 11 20:00:21.656251 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.656206    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5" event={"ID":"4699764c-395c-4b7b-b7ae-ff752ae93302","Type":"ContainerDied","Data":"f9dd2106d178abefc5cf8a2df9278831266a2ba9daa60c8087f6285142e349aa"}
Sep 11 20:00:21.658255 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.658233    2571 generic.go:356] "Generic (PLEG): container finished" podID="e558950d-0027-4a14-9600-df9dbb6e86dc" containerID="97441a64d791f01aad43d88ef8a93e4b10f617ef0bcf4d71729bcafd55ac34a9" exitCode=0
Sep 11 20:00:21.658900 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.658415    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr" event={"ID":"e558950d-0027-4a14-9600-df9dbb6e86dc","Type":"ContainerDied","Data":"97441a64d791f01aad43d88ef8a93e4b10f617ef0bcf4d71729bcafd55ac34a9"}
Sep 11 20:00:21.806381 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.805995    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 20:00:21.849719 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.847076    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-model-cache\") pod \"4699764c-395c-4b7b-b7ae-ff752ae93302\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") "
Sep 11 20:00:21.851287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.850419    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-home\") pod \"4699764c-395c-4b7b-b7ae-ff752ae93302\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") "
Sep 11 20:00:21.851287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.850479    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4699764c-395c-4b7b-b7ae-ff752ae93302-tls-certs\") pod \"4699764c-395c-4b7b-b7ae-ff752ae93302\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") "
Sep 11 20:00:21.851287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.850518    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-tmp-dir\") pod \"4699764c-395c-4b7b-b7ae-ff752ae93302\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") "
Sep 11 20:00:21.851287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.850558    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-dshm\") pod \"4699764c-395c-4b7b-b7ae-ff752ae93302\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") "
Sep 11 20:00:21.851287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.850598    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbww5\" (UniqueName: \"kubernetes.io/projected/4699764c-395c-4b7b-b7ae-ff752ae93302-kube-api-access-nbww5\") pod \"4699764c-395c-4b7b-b7ae-ff752ae93302\" (UID: \"4699764c-395c-4b7b-b7ae-ff752ae93302\") "
Sep 11 20:00:21.853946 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.850133    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-model-cache" (OuterVolumeSpecName: "model-cache") pod "4699764c-395c-4b7b-b7ae-ff752ae93302" (UID: "4699764c-395c-4b7b-b7ae-ff752ae93302"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.853946 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.849853    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 20:00:21.853946 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.851879    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-home" (OuterVolumeSpecName: "home") pod "4699764c-395c-4b7b-b7ae-ff752ae93302" (UID: "4699764c-395c-4b7b-b7ae-ff752ae93302"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.853946 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.852054    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4699764c-395c-4b7b-b7ae-ff752ae93302" (UID: "4699764c-395c-4b7b-b7ae-ff752ae93302"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.865953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.865918    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4699764c-395c-4b7b-b7ae-ff752ae93302-kube-api-access-nbww5" (OuterVolumeSpecName: "kube-api-access-nbww5") pod "4699764c-395c-4b7b-b7ae-ff752ae93302" (UID: "4699764c-395c-4b7b-b7ae-ff752ae93302"). InnerVolumeSpecName "kube-api-access-nbww5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:00:21.869863 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.869823    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-dshm" (OuterVolumeSpecName: "dshm") pod "4699764c-395c-4b7b-b7ae-ff752ae93302" (UID: "4699764c-395c-4b7b-b7ae-ff752ae93302"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.880231 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.880187    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4699764c-395c-4b7b-b7ae-ff752ae93302-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4699764c-395c-4b7b-b7ae-ff752ae93302" (UID: "4699764c-395c-4b7b-b7ae-ff752ae93302"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:00:21.906930 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.906901    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 20:00:21.910197 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.910174    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 20:00:21.951506 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951474    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-home\") pod \"162965eb-da36-444b-9305-b27bb7ac071c\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") "
Sep 11 20:00:21.951692 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951520    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njc69\" (UniqueName: \"kubernetes.io/projected/e558950d-0027-4a14-9600-df9dbb6e86dc-kube-api-access-njc69\") pod \"e558950d-0027-4a14-9600-df9dbb6e86dc\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") "
Sep 11 20:00:21.951692 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951548    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162965eb-da36-444b-9305-b27bb7ac071c-tls-certs\") pod \"162965eb-da36-444b-9305-b27bb7ac071c\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") "
Sep 11 20:00:21.951692 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951582    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-model-cache\") pod \"162965eb-da36-444b-9305-b27bb7ac071c\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") "
Sep 11 20:00:21.951692 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951652    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-home\") pod \"e558950d-0027-4a14-9600-df9dbb6e86dc\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") "
Sep 11 20:00:21.951692 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951677    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-dshm\") pod \"e558950d-0027-4a14-9600-df9dbb6e86dc\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") "
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951727    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-model-cache\") pod \"e558950d-0027-4a14-9600-df9dbb6e86dc\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") "
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951734    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-home" (OuterVolumeSpecName: "home") pod "162965eb-da36-444b-9305-b27bb7ac071c" (UID: "162965eb-da36-444b-9305-b27bb7ac071c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951752    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e558950d-0027-4a14-9600-df9dbb6e86dc-tls-certs\") pod \"e558950d-0027-4a14-9600-df9dbb6e86dc\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") "
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951813    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-tmp-dir\") pod \"22db71ec-b9b9-4de7-91b7-be46436e8270\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") "
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951845    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqhsp\" (UniqueName: \"kubernetes.io/projected/22db71ec-b9b9-4de7-91b7-be46436e8270-kube-api-access-hqhsp\") pod \"22db71ec-b9b9-4de7-91b7-be46436e8270\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") "
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951873    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-dshm\") pod \"22db71ec-b9b9-4de7-91b7-be46436e8270\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") "
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951907    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-home\") pod \"22db71ec-b9b9-4de7-91b7-be46436e8270\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") "
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951905    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-home" (OuterVolumeSpecName: "home") pod "e558950d-0027-4a14-9600-df9dbb6e86dc" (UID: "e558950d-0027-4a14-9600-df9dbb6e86dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951937    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-model-cache\") pod \"22db71ec-b9b9-4de7-91b7-be46436e8270\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") "
Sep 11 20:00:21.951964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951966    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q696w\" (UniqueName: \"kubernetes.io/projected/162965eb-da36-444b-9305-b27bb7ac071c-kube-api-access-q696w\") pod \"162965eb-da36-444b-9305-b27bb7ac071c\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") "
Sep 11 20:00:21.952503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.951990    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-tmp-dir\") pod \"162965eb-da36-444b-9305-b27bb7ac071c\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") "
Sep 11 20:00:21.952503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952022    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-tmp-dir\") pod \"e558950d-0027-4a14-9600-df9dbb6e86dc\" (UID: \"e558950d-0027-4a14-9600-df9dbb6e86dc\") "
Sep 11 20:00:21.952503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952058    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-model-cache" (OuterVolumeSpecName: "model-cache") pod "162965eb-da36-444b-9305-b27bb7ac071c" (UID: "162965eb-da36-444b-9305-b27bb7ac071c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.952503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952065    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db71ec-b9b9-4de7-91b7-be46436e8270-tls-certs\") pod \"22db71ec-b9b9-4de7-91b7-be46436e8270\" (UID: \"22db71ec-b9b9-4de7-91b7-be46436e8270\") "
Sep 11 20:00:21.952503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952096    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-dshm\") pod \"162965eb-da36-444b-9305-b27bb7ac071c\" (UID: \"162965eb-da36-444b-9305-b27bb7ac071c\") "
Sep 11 20:00:21.952503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952233    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e558950d-0027-4a14-9600-df9dbb6e86dc" (UID: "e558950d-0027-4a14-9600-df9dbb6e86dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952538    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952560    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952574    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4699764c-395c-4b7b-b7ae-ff752ae93302-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952589    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952603    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952618    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbww5\" (UniqueName: \"kubernetes.io/projected/4699764c-395c-4b7b-b7ae-ff752ae93302-kube-api-access-nbww5\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952632    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952645    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952658    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4699764c-395c-4b7b-b7ae-ff752ae93302-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.952815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.952668    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:21.953500 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.953477    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22db71ec-b9b9-4de7-91b7-be46436e8270" (UID: "22db71ec-b9b9-4de7-91b7-be46436e8270"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.954998 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.954809    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/162965eb-da36-444b-9305-b27bb7ac071c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "162965eb-da36-444b-9305-b27bb7ac071c" (UID: "162965eb-da36-444b-9305-b27bb7ac071c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:00:21.955371 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.955188    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e558950d-0027-4a14-9600-df9dbb6e86dc" (UID: "e558950d-0027-4a14-9600-df9dbb6e86dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.955371 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.955248    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-dshm" (OuterVolumeSpecName: "dshm") pod "e558950d-0027-4a14-9600-df9dbb6e86dc" (UID: "e558950d-0027-4a14-9600-df9dbb6e86dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.955529 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.955448    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-home" (OuterVolumeSpecName: "home") pod "22db71ec-b9b9-4de7-91b7-be46436e8270" (UID: "22db71ec-b9b9-4de7-91b7-be46436e8270"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.955529 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.955469    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e558950d-0027-4a14-9600-df9dbb6e86dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e558950d-0027-4a14-9600-df9dbb6e86dc" (UID: "e558950d-0027-4a14-9600-df9dbb6e86dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:00:21.955910 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.955701    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "162965eb-da36-444b-9305-b27bb7ac071c" (UID: "162965eb-da36-444b-9305-b27bb7ac071c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.956090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.956031    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22db71ec-b9b9-4de7-91b7-be46436e8270-kube-api-access-hqhsp" (OuterVolumeSpecName: "kube-api-access-hqhsp") pod "22db71ec-b9b9-4de7-91b7-be46436e8270" (UID: "22db71ec-b9b9-4de7-91b7-be46436e8270"). InnerVolumeSpecName "kube-api-access-hqhsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:00:21.956090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.956039    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-model-cache" (OuterVolumeSpecName: "model-cache") pod "22db71ec-b9b9-4de7-91b7-be46436e8270" (UID: "22db71ec-b9b9-4de7-91b7-be46436e8270"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.956752 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.956495    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e558950d-0027-4a14-9600-df9dbb6e86dc-kube-api-access-njc69" (OuterVolumeSpecName: "kube-api-access-njc69") pod "e558950d-0027-4a14-9600-df9dbb6e86dc" (UID: "e558950d-0027-4a14-9600-df9dbb6e86dc"). InnerVolumeSpecName "kube-api-access-njc69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:00:21.956870 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.956846    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-dshm" (OuterVolumeSpecName: "dshm") pod "162965eb-da36-444b-9305-b27bb7ac071c" (UID: "162965eb-da36-444b-9305-b27bb7ac071c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:21.957800 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.957765    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22db71ec-b9b9-4de7-91b7-be46436e8270-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22db71ec-b9b9-4de7-91b7-be46436e8270" (UID: "22db71ec-b9b9-4de7-91b7-be46436e8270"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:00:21.957903 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.957839    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162965eb-da36-444b-9305-b27bb7ac071c-kube-api-access-q696w" (OuterVolumeSpecName: "kube-api-access-q696w") pod "162965eb-da36-444b-9305-b27bb7ac071c" (UID: "162965eb-da36-444b-9305-b27bb7ac071c"). InnerVolumeSpecName "kube-api-access-q696w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:00:21.958132 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:21.958081    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-dshm" (OuterVolumeSpecName: "dshm") pod "22db71ec-b9b9-4de7-91b7-be46436e8270" (UID: "22db71ec-b9b9-4de7-91b7-be46436e8270"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:22.053196 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053168    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njc69\" (UniqueName: \"kubernetes.io/projected/e558950d-0027-4a14-9600-df9dbb6e86dc-kube-api-access-njc69\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053196 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053201    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162965eb-da36-444b-9305-b27bb7ac071c-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053217    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053230    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e558950d-0027-4a14-9600-df9dbb6e86dc-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053241    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053250    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqhsp\" (UniqueName: \"kubernetes.io/projected/22db71ec-b9b9-4de7-91b7-be46436e8270-kube-api-access-hqhsp\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053261    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053272    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053282    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db71ec-b9b9-4de7-91b7-be46436e8270-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053293    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q696w\" (UniqueName: \"kubernetes.io/projected/162965eb-da36-444b-9305-b27bb7ac071c-kube-api-access-q696w\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053306    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053320    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e558950d-0027-4a14-9600-df9dbb6e86dc-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053328    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db71ec-b9b9-4de7-91b7-be46436e8270-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.053382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.053338    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162965eb-da36-444b-9305-b27bb7ac071c-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:22.665546 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.665518    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"
Sep 11 20:00:22.665975 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.665555    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6" event={"ID":"22db71ec-b9b9-4de7-91b7-be46436e8270","Type":"ContainerDied","Data":"809f1af66255624d0437ec396ff11cc45c50100b3d5960c139717ac565f61e95"}
Sep 11 20:00:22.665975 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.665612    2571 scope.go:117] "RemoveContainer" containerID="ab72ebc5262a2c4809912c1d8b8e14730e4d6088e0d70bdb25e01db1e677610e"
Sep 11 20:00:22.667032 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.667011    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5" event={"ID":"4699764c-395c-4b7b-b7ae-ff752ae93302","Type":"ContainerDied","Data":"b20a956c53a7985f075ca558b2f8dd7ca285efb2b3e93adb36500b8d6adfd733"}
Sep 11 20:00:22.667145 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.667039    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"
Sep 11 20:00:22.668603 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.668576    2571 generic.go:356] "Generic (PLEG): container finished" podID="162965eb-da36-444b-9305-b27bb7ac071c" containerID="8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d" exitCode=0
Sep 11 20:00:22.668692 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.668614    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"
Sep 11 20:00:22.668768 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.668672    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm" event={"ID":"162965eb-da36-444b-9305-b27bb7ac071c","Type":"ContainerDied","Data":"8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d"}
Sep 11 20:00:22.668768 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.668723    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm" event={"ID":"162965eb-da36-444b-9305-b27bb7ac071c","Type":"ContainerDied","Data":"f7f3fb2f86ac1ab5a6c574c89ce426297d0ed4990552688e4fced306fe5cdba7"}
Sep 11 20:00:22.670451 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.670431    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr" event={"ID":"e558950d-0027-4a14-9600-df9dbb6e86dc","Type":"ContainerDied","Data":"43873b05c0bb52c4611f76ea625557a66b5a0e738bc7422e859a5ce7ef3dbbe1"}
Sep 11 20:00:22.670545 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.670457    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"
Sep 11 20:00:22.676435 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.676419    2571 scope.go:117] "RemoveContainer" containerID="f9dd2106d178abefc5cf8a2df9278831266a2ba9daa60c8087f6285142e349aa"
Sep 11 20:00:22.685826 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.685810    2571 scope.go:117] "RemoveContainer" containerID="8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d"
Sep 11 20:00:22.692958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.692932    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"]
Sep 11 20:00:22.694343 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.694326    2571 scope.go:117] "RemoveContainer" containerID="8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d"
Sep 11 20:00:22.694602 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:00:22.694584    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d\": container with ID starting with 8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d not found: ID does not exist" containerID="8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d"
Sep 11 20:00:22.694645 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.694611    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d"} err="failed to get container status \"8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d\": rpc error: code = NotFound desc = could not find container \"8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d\": container with ID starting with 8350f4f5e263df808ed99524c261b11e4d756225189149535db9dfb2f7399e6d not found: ID does not exist"
Sep 11 20:00:22.694645 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.694629    2571 scope.go:117] "RemoveContainer" containerID="97441a64d791f01aad43d88ef8a93e4b10f617ef0bcf4d71729bcafd55ac34a9"
Sep 11 20:00:22.698577 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.698556    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/premium-w0-4b032e-kserve-58dcb89dfd-ncvk6"]
Sep 11 20:00:22.709614 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.709595    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"]
Sep 11 20:00:22.715237 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.715217    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/embedding-w0-4b032e-kserve-5b7c459d7d-924f5"]
Sep 11 20:00:22.734208 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.734178    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"]
Sep 11 20:00:22.736525 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.736505    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/distinct-2-w0-4b032e-kserve-6d5b895998-9kxtr"]
Sep 11 20:00:22.747625 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.747601    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"]
Sep 11 20:00:22.754245 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:22.754225    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b032e/simulator-w0-4b032e-kserve-64d4d588c5-nscpm"]
Sep 11 20:00:23.873882 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:23.873849    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="162965eb-da36-444b-9305-b27bb7ac071c" path="/var/lib/kubelet/pods/162965eb-da36-444b-9305-b27bb7ac071c/volumes"
Sep 11 20:00:23.874248 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:23.874235    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22db71ec-b9b9-4de7-91b7-be46436e8270" path="/var/lib/kubelet/pods/22db71ec-b9b9-4de7-91b7-be46436e8270/volumes"
Sep 11 20:00:23.874595 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:23.874581    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4699764c-395c-4b7b-b7ae-ff752ae93302" path="/var/lib/kubelet/pods/4699764c-395c-4b7b-b7ae-ff752ae93302/volumes"
Sep 11 20:00:23.875223 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:23.875209    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e558950d-0027-4a14-9600-df9dbb6e86dc" path="/var/lib/kubelet/pods/e558950d-0027-4a14-9600-df9dbb6e86dc/volumes"
Sep 11 20:00:31.451266 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:31.451215    2571 patch_prober.go:28] interesting pod/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 20:00:31.451802 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:31.451279    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 20:00:31.660838 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:31.660806    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:00:31.661050 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:31.661034    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:00:32.750659 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:32.750574    2571 generic.go:356] "Generic (PLEG): container finished" podID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerID="aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d" exitCode=0
Sep 11 20:00:32.751041 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:32.750657    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" event={"ID":"ddd493de-1d79-427c-aa36-b2f2725a4a44","Type":"ContainerDied","Data":"aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d"}
Sep 11 20:00:32.751188 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:32.751170    2571 scope.go:117] "RemoveContainer" containerID="aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d"
Sep 11 20:00:32.752497 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:32.752468    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:00:32.752609 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:32.752553    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:00:32.752609 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:32.752604    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:00:33.763578 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:33.763544    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" event={"ID":"ddd493de-1d79-427c-aa36-b2f2725a4a44","Type":"ContainerStarted","Data":"41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482"}
Sep 11 20:00:33.851667 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:33.851622    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 20:00:33.851863 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:33.851841    2571 patch_prober.go:28] interesting pod/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 20:00:33.851945 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:33.851901    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 20:00:34.851509 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:34.851466    2571 patch_prober.go:28] interesting pod/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 20:00:34.851888 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:34.851549    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 20:00:35.851179 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:35.851141    2571 patch_prober.go:28] interesting pod/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 20:00:35.851347 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:35.851195    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 20:00:36.851060 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:36.851017    2571 patch_prober.go:28] interesting pod/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 20:00:36.851483 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:36.851081    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 20:00:37.851832 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:37.851792    2571 patch_prober.go:28] interesting pod/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 11 20:00:37.852252 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:37.851864    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 11 20:00:38.856781 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:38.856746    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 20:00:38.857268 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:38.857136    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 20:00:38.858004 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:38.857985    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 20:00:46.734752 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.734668    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"]
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735024    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22db71ec-b9b9-4de7-91b7-be46436e8270" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735035    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="22db71ec-b9b9-4de7-91b7-be46436e8270" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735044    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4699764c-395c-4b7b-b7ae-ff752ae93302" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735049    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4699764c-395c-4b7b-b7ae-ff752ae93302" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735063    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e558950d-0027-4a14-9600-df9dbb6e86dc" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735068    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e558950d-0027-4a14-9600-df9dbb6e86dc" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735074    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="162965eb-da36-444b-9305-b27bb7ac071c" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735079    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="162965eb-da36-444b-9305-b27bb7ac071c" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735161    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="22db71ec-b9b9-4de7-91b7-be46436e8270" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735171    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e558950d-0027-4a14-9600-df9dbb6e86dc" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735178    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4699764c-395c-4b7b-b7ae-ff752ae93302" containerName="main"
Sep 11 20:00:46.735202 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.735185    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="162965eb-da36-444b-9305-b27bb7ac071c" containerName="main"
Sep 11 20:00:46.738440 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.738416    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"
Sep 11 20:00:46.741355 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.741328    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:00:46.741466 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.741328    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:00:46.747294 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.747270    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"]
Sep 11 20:00:46.893171 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.893136    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e00243-1b27-451b-94ab-5baf485758eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8\" (UID: \"67e00243-1b27-451b-94ab-5baf485758eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"
Sep 11 20:00:46.994692 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.994590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e00243-1b27-451b-94ab-5baf485758eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8\" (UID: \"67e00243-1b27-451b-94ab-5baf485758eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"
Sep 11 20:00:46.995306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:46.995284    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e00243-1b27-451b-94ab-5baf485758eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8\" (UID: \"67e00243-1b27-451b-94ab-5baf485758eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"
Sep 11 20:00:47.050231 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:47.050197    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"
Sep 11 20:00:47.324754 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:47.324726    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"]
Sep 11 20:00:47.325725 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:00:47.325697    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e00243_1b27_451b_94ab_5baf485758eb.slice/crio-f6b9b2646b44797ba69d186e3892a8ccda5cf291d9711ffb216ed0d8e391e246 WatchSource:0}: Error finding container f6b9b2646b44797ba69d186e3892a8ccda5cf291d9711ffb216ed0d8e391e246: Status 404 returned error can't find the container with id f6b9b2646b44797ba69d186e3892a8ccda5cf291d9711ffb216ed0d8e391e246
Sep 11 20:00:47.844592 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:47.844560    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8" event={"ID":"67e00243-1b27-451b-94ab-5baf485758eb","Type":"ContainerStarted","Data":"25c52bde824049a330f5d835f5553c9bfe8452879939d800bdeeed60def00f0a"}
Sep 11 20:00:47.844999 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:47.844615    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8" event={"ID":"67e00243-1b27-451b-94ab-5baf485758eb","Type":"ContainerStarted","Data":"f6b9b2646b44797ba69d186e3892a8ccda5cf291d9711ffb216ed0d8e391e246"}
Sep 11 20:00:47.861493 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:47.861436    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8" podStartSLOduration=1.86141762 podStartE2EDuration="1.86141762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:47.858714808 +0000 UTC m=+1884.534651727" watchObservedRunningTime="2026-09-11 20:00:47.86141762 +0000 UTC m=+1884.537354541"
Sep 11 20:00:48.851424 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:48.851385    2571 generic.go:356] "Generic (PLEG): container finished" podID="67e00243-1b27-451b-94ab-5baf485758eb" containerID="25c52bde824049a330f5d835f5553c9bfe8452879939d800bdeeed60def00f0a" exitCode=0
Sep 11 20:00:48.851804 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:48.851468    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8" event={"ID":"67e00243-1b27-451b-94ab-5baf485758eb","Type":"ContainerDied","Data":"25c52bde824049a330f5d835f5553c9bfe8452879939d800bdeeed60def00f0a"}
Sep 11 20:00:49.974989 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:49.974967    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"
Sep 11 20:00:50.125205 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:50.125177    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e00243-1b27-451b-94ab-5baf485758eb-maas-api-service-ca\") pod \"67e00243-1b27-451b-94ab-5baf485758eb\" (UID: \"67e00243-1b27-451b-94ab-5baf485758eb\") "
Sep 11 20:00:50.125487 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:50.125465    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e00243-1b27-451b-94ab-5baf485758eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67e00243-1b27-451b-94ab-5baf485758eb" (UID: "67e00243-1b27-451b-94ab-5baf485758eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:00:50.226385 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:50.226358    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e00243-1b27-451b-94ab-5baf485758eb-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:00:50.864456 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:50.864424    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"
Sep 11 20:00:50.864636 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:50.864462    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8" event={"ID":"67e00243-1b27-451b-94ab-5baf485758eb","Type":"ContainerDied","Data":"f6b9b2646b44797ba69d186e3892a8ccda5cf291d9711ffb216ed0d8e391e246"}
Sep 11 20:00:50.864636 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:50.864520    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b9b2646b44797ba69d186e3892a8ccda5cf291d9711ffb216ed0d8e391e246"
Sep 11 20:00:56.750663 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:00:56.750628    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:01:02.453027 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.452990    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"]
Sep 11 20:01:02.460227 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.460201    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e7cdd6-29819280-cqpnk"]
Sep 11 20:01:02.489811 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.489773    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"]
Sep 11 20:01:02.490025 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.489986    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg" podUID="d157afbf-dbea-41a9-8035-8f6140d02f9a" containerName="payload-pre-processing" containerID="cri-o://56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf" gracePeriod=30
Sep 11 20:01:02.649768 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.649735    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"]
Sep 11 20:01:02.652065 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.652042    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e7cdd6-jvnr8"]
Sep 11 20:01:02.742184 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.742155    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 20:01:02.841967 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.841934    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d157afbf-dbea-41a9-8035-8f6140d02f9a-plugins-config-volume\") pod \"d157afbf-dbea-41a9-8035-8f6140d02f9a\" (UID: \"d157afbf-dbea-41a9-8035-8f6140d02f9a\") "
Sep 11 20:01:02.842170 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.842023    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddh4w\" (UniqueName: \"kubernetes.io/projected/d157afbf-dbea-41a9-8035-8f6140d02f9a-kube-api-access-ddh4w\") pod \"d157afbf-dbea-41a9-8035-8f6140d02f9a\" (UID: \"d157afbf-dbea-41a9-8035-8f6140d02f9a\") "
Sep 11 20:01:02.842330 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.842305    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d157afbf-dbea-41a9-8035-8f6140d02f9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d157afbf-dbea-41a9-8035-8f6140d02f9a" (UID: "d157afbf-dbea-41a9-8035-8f6140d02f9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:02.844076 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.844052    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d157afbf-dbea-41a9-8035-8f6140d02f9a-kube-api-access-ddh4w" (OuterVolumeSpecName: "kube-api-access-ddh4w") pod "d157afbf-dbea-41a9-8035-8f6140d02f9a" (UID: "d157afbf-dbea-41a9-8035-8f6140d02f9a"). InnerVolumeSpecName "kube-api-access-ddh4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:02.934014 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.933976    2571 generic.go:356] "Generic (PLEG): container finished" podID="d157afbf-dbea-41a9-8035-8f6140d02f9a" containerID="56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf" exitCode=0
Sep 11 20:01:02.934189 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.934024    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"
Sep 11 20:01:02.934189 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.934057    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg" event={"ID":"d157afbf-dbea-41a9-8035-8f6140d02f9a","Type":"ContainerDied","Data":"56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf"}
Sep 11 20:01:02.934189 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.934124    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg" event={"ID":"d157afbf-dbea-41a9-8035-8f6140d02f9a","Type":"ContainerDied","Data":"9db7d32c352650e328f265088b2d2f49dfb0e43507fc000ccdf130b1a59a9eb2"}
Sep 11 20:01:02.934189 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.934143    2571 scope.go:117] "RemoveContainer" containerID="56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf"
Sep 11 20:01:02.942729 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.942696    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d157afbf-dbea-41a9-8035-8f6140d02f9a-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:02.942729 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.942729    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddh4w\" (UniqueName: \"kubernetes.io/projected/d157afbf-dbea-41a9-8035-8f6140d02f9a-kube-api-access-ddh4w\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:02.944990 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.944972    2571 scope.go:117] "RemoveContainer" containerID="56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf"
Sep 11 20:01:02.945496 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:02.945469    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf\": container with ID starting with 56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf not found: ID does not exist" containerID="56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf"
Sep 11 20:01:02.945589 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.945502    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf"} err="failed to get container status \"56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf\": rpc error: code = NotFound desc = could not find container \"56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf\": container with ID starting with 56da302f5cb9c7f4296dbbb42d69dbf33d7a20b0f794eadc30d08b324e833baf not found: ID does not exist"
Sep 11 20:01:02.960072 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.960045    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"]
Sep 11 20:01:02.970284 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:02.970257    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e7cdd6-7cdccf8786-jwmbg"]
Sep 11 20:01:03.873367 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:03.873319    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e00243-1b27-451b-94ab-5baf485758eb" path="/var/lib/kubelet/pods/67e00243-1b27-451b-94ab-5baf485758eb/volumes"
Sep 11 20:01:03.873834 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:03.873709    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6acb00c7-b7e0-432d-9fca-110732e48033" path="/var/lib/kubelet/pods/6acb00c7-b7e0-432d-9fca-110732e48033/volumes"
Sep 11 20:01:03.874021 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:03.874007    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d157afbf-dbea-41a9-8035-8f6140d02f9a" path="/var/lib/kubelet/pods/d157afbf-dbea-41a9-8035-8f6140d02f9a/volumes"
Sep 11 20:01:09.216490 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.216448    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"]
Sep 11 20:01:09.216973 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.216732    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy" containerID="cri-o://41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482" gracePeriod=30
Sep 11 20:01:09.496697 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.496605    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"]
Sep 11 20:01:09.497044 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.497024    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e00243-1b27-451b-94ab-5baf485758eb" containerName="revoke-keys"
Sep 11 20:01:09.497163 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.497047    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e00243-1b27-451b-94ab-5baf485758eb" containerName="revoke-keys"
Sep 11 20:01:09.497163 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.497068    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d157afbf-dbea-41a9-8035-8f6140d02f9a" containerName="payload-pre-processing"
Sep 11 20:01:09.497163 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.497079    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d157afbf-dbea-41a9-8035-8f6140d02f9a" containerName="payload-pre-processing"
Sep 11 20:01:09.497328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.497204    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e00243-1b27-451b-94ab-5baf485758eb" containerName="revoke-keys"
Sep 11 20:01:09.497328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.497221    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d157afbf-dbea-41a9-8035-8f6140d02f9a" containerName="payload-pre-processing"
Sep 11 20:01:09.501757 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.501735    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"
Sep 11 20:01:09.504712 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.504688    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:01:09.504934 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.504911    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:01:09.512471 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.512437    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"]
Sep 11 20:01:09.600368 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.600334    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863f583f-2242-439b-aaf7-cacd98ed98a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn\" (UID: \"863f583f-2242-439b-aaf7-cacd98ed98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"
Sep 11 20:01:09.700955 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.700915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863f583f-2242-439b-aaf7-cacd98ed98a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn\" (UID: \"863f583f-2242-439b-aaf7-cacd98ed98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"
Sep 11 20:01:09.701613 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.701591    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863f583f-2242-439b-aaf7-cacd98ed98a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn\" (UID: \"863f583f-2242-439b-aaf7-cacd98ed98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"
Sep 11 20:01:09.814058 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:09.813980    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"
Sep 11 20:01:10.089176 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:10.089149    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"]
Sep 11 20:01:10.091165 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:01:10.091140    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863f583f_2242_439b_aaf7_cacd98ed98a6.slice/crio-c0b663b51c2ddbca78cdc78b3bb3b0988814d5ee0d250c63d25b4208b5aa09fc WatchSource:0}: Error finding container c0b663b51c2ddbca78cdc78b3bb3b0988814d5ee0d250c63d25b4208b5aa09fc: Status 404 returned error can't find the container with id c0b663b51c2ddbca78cdc78b3bb3b0988814d5ee0d250c63d25b4208b5aa09fc
Sep 11 20:01:10.981257 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:10.981218    2571 generic.go:356] "Generic (PLEG): container finished" podID="863f583f-2242-439b-aaf7-cacd98ed98a6" containerID="9c23b4b964cb8a03a9a85bc8a2b1d205201bdfa1d72cdaea0aa99ebe4b2e0b21" exitCode=0
Sep 11 20:01:10.981653 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:10.981304    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn" event={"ID":"863f583f-2242-439b-aaf7-cacd98ed98a6","Type":"ContainerDied","Data":"9c23b4b964cb8a03a9a85bc8a2b1d205201bdfa1d72cdaea0aa99ebe4b2e0b21"}
Sep 11 20:01:10.981653 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:10.981371    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn" event={"ID":"863f583f-2242-439b-aaf7-cacd98ed98a6","Type":"ContainerStarted","Data":"c0b663b51c2ddbca78cdc78b3bb3b0988814d5ee0d250c63d25b4208b5aa09fc"}
Sep 11 20:01:12.111072 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:12.111048    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"
Sep 11 20:01:12.226571 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:12.226539    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863f583f-2242-439b-aaf7-cacd98ed98a6-maas-api-service-ca\") pod \"863f583f-2242-439b-aaf7-cacd98ed98a6\" (UID: \"863f583f-2242-439b-aaf7-cacd98ed98a6\") "
Sep 11 20:01:12.226854 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:12.226832    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863f583f-2242-439b-aaf7-cacd98ed98a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "863f583f-2242-439b-aaf7-cacd98ed98a6" (UID: "863f583f-2242-439b-aaf7-cacd98ed98a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:12.327539 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:12.327451    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/863f583f-2242-439b-aaf7-cacd98ed98a6-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:12.993753 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:12.993714    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn" event={"ID":"863f583f-2242-439b-aaf7-cacd98ed98a6","Type":"ContainerDied","Data":"c0b663b51c2ddbca78cdc78b3bb3b0988814d5ee0d250c63d25b4208b5aa09fc"}
Sep 11 20:01:12.993753 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:12.993741    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"
Sep 11 20:01:12.993970 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:12.993776    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0b663b51c2ddbca78cdc78b3bb3b0988814d5ee0d250c63d25b4208b5aa09fc"
Sep 11 20:01:14.462676 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.462655    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 20:01:14.546544 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546468    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-credential-socket\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546544 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546522    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-certs\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546724 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546568    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-data\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546724 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546599    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-envoy\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546724 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546652    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-socket\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546724 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546709    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:14.546953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546713    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr6wz\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-kube-api-access-pr6wz\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546807    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-token\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546849    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:14.546953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546861    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-podinfo\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546868    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-data" (OuterVolumeSpecName: "istio-data") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:14.546953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546892    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd493de-1d79-427c-aa36-b2f2725a4a44-istiod-ca-cert\") pod \"ddd493de-1d79-427c-aa36-b2f2725a4a44\" (UID: \"ddd493de-1d79-427c-aa36-b2f2725a4a44\") "
Sep 11 20:01:14.546953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.546948    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:14.547356 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.547211    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd493de-1d79-427c-aa36-b2f2725a4a44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:14.547356 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.547227    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:14.547356 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.547249    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:14.547356 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.547263    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:14.547356 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.547275    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:14.548797 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.548770    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 20:01:14.548896 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.548794    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:14.548896 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.548876    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:14.548896 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.548888    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-kube-api-access-pr6wz" (OuterVolumeSpecName: "kube-api-access-pr6wz") pod "ddd493de-1d79-427c-aa36-b2f2725a4a44" (UID: "ddd493de-1d79-427c-aa36-b2f2725a4a44"). InnerVolumeSpecName "kube-api-access-pr6wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:14.648453 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.648421    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:14.648453 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.648452    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr6wz\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-kube-api-access-pr6wz\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:14.648453 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.648462    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:14.648625 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.648471    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd493de-1d79-427c-aa36-b2f2725a4a44-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:14.648625 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:14.648480    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd493de-1d79-427c-aa36-b2f2725a4a44-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:15.006312 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.006284    2571 generic.go:356] "Generic (PLEG): container finished" podID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerID="41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482" exitCode=0
Sep 11 20:01:15.006485 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.006326    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"
Sep 11 20:01:15.006485 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.006365    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" event={"ID":"ddd493de-1d79-427c-aa36-b2f2725a4a44","Type":"ContainerDied","Data":"41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482"}
Sep 11 20:01:15.006485 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.006427    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6" event={"ID":"ddd493de-1d79-427c-aa36-b2f2725a4a44","Type":"ContainerDied","Data":"175d21989938a83a7be711709108d0e6a0104f2a1d36451f36586a4254acd909"}
Sep 11 20:01:15.006485 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.006450    2571 scope.go:117] "RemoveContainer" containerID="41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482"
Sep 11 20:01:15.016425 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.016411    2571 scope.go:117] "RemoveContainer" containerID="aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d"
Sep 11 20:01:15.024566 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.024550    2571 scope.go:117] "RemoveContainer" containerID="41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482"
Sep 11 20:01:15.024828 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:15.024808    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482\": container with ID starting with 41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482 not found: ID does not exist" containerID="41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482"
Sep 11 20:01:15.024893 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.024840    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482"} err="failed to get container status \"41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482\": rpc error: code = NotFound desc = could not find container \"41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482\": container with ID starting with 41105900c6b95182568ba3733be618bd4e4cfd1ab7122425778db58b3e78a482 not found: ID does not exist"
Sep 11 20:01:15.024893 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.024865    2571 scope.go:117] "RemoveContainer" containerID="aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d"
Sep 11 20:01:15.025134 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:15.025115    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d\": container with ID starting with aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d not found: ID does not exist" containerID="aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d"
Sep 11 20:01:15.025184 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.025140    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d"} err="failed to get container status \"aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d\": rpc error: code = NotFound desc = could not find container \"aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d\": container with ID starting with aa6c690220c27dd43d2f2bca9b39670e92f2cae9e821e02019530a6d3a516b2d not found: ID does not exist"
Sep 11 20:01:15.029852 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.029833    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"]
Sep 11 20:01:15.033300 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.033278    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e7cdd6-openshift-default-746dbfb5f8-lnrn6"]
Sep 11 20:01:15.873507 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:15.873473    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" path="/var/lib/kubelet/pods/ddd493de-1d79-427c-aa36-b2f2725a4a44/volumes"
Sep 11 20:01:17.363525 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363487    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"]
Sep 11 20:01:17.363894 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363834    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy"
Sep 11 20:01:17.363894 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363844    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy"
Sep 11 20:01:17.363894 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363852    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy"
Sep 11 20:01:17.363894 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363858    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy"
Sep 11 20:01:17.363894 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363874    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863f583f-2242-439b-aaf7-cacd98ed98a6" containerName="revoke-keys"
Sep 11 20:01:17.363894 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363879    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="863f583f-2242-439b-aaf7-cacd98ed98a6" containerName="revoke-keys"
Sep 11 20:01:17.364145 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363934    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="863f583f-2242-439b-aaf7-cacd98ed98a6" containerName="revoke-keys"
Sep 11 20:01:17.364145 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.363943    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy"
Sep 11 20:01:17.368485 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.368468    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"
Sep 11 20:01:17.372441 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.372399    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:01:17.372441 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.372407    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:01:17.374677 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.374650    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"]
Sep 11 20:01:17.471479 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.471435    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1cbb2-1878-4aa0-a65d-e07c7bcc3902-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f\" (UID: \"56e1cbb2-1878-4aa0-a65d-e07c7bcc3902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"
Sep 11 20:01:17.572331 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.572292    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1cbb2-1878-4aa0-a65d-e07c7bcc3902-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f\" (UID: \"56e1cbb2-1878-4aa0-a65d-e07c7bcc3902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"
Sep 11 20:01:17.572923 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.572902    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1cbb2-1878-4aa0-a65d-e07c7bcc3902-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f\" (UID: \"56e1cbb2-1878-4aa0-a65d-e07c7bcc3902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"
Sep 11 20:01:17.680684 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.680650    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"
Sep 11 20:01:17.955325 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:17.955298    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"]
Sep 11 20:01:17.956871 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:01:17.956845    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e1cbb2_1878_4aa0_a65d_e07c7bcc3902.slice/crio-b21db5ad5435f1110a6783056eba12db710132fa2f5524621c0cc2c8863bb244 WatchSource:0}: Error finding container b21db5ad5435f1110a6783056eba12db710132fa2f5524621c0cc2c8863bb244: Status 404 returned error can't find the container with id b21db5ad5435f1110a6783056eba12db710132fa2f5524621c0cc2c8863bb244
Sep 11 20:01:18.025986 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:18.025955    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f" event={"ID":"56e1cbb2-1878-4aa0-a65d-e07c7bcc3902","Type":"ContainerStarted","Data":"b21db5ad5435f1110a6783056eba12db710132fa2f5524621c0cc2c8863bb244"}
Sep 11 20:01:19.033684 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:19.033647    2571 generic.go:356] "Generic (PLEG): container finished" podID="56e1cbb2-1878-4aa0-a65d-e07c7bcc3902" containerID="411905526ec888ac42710022c31c6b80646450cf4c6e6e97451f4ddc4120bec9" exitCode=0
Sep 11 20:01:19.033684 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:19.033690    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f" event={"ID":"56e1cbb2-1878-4aa0-a65d-e07c7bcc3902","Type":"ContainerDied","Data":"411905526ec888ac42710022c31c6b80646450cf4c6e6e97451f4ddc4120bec9"}
Sep 11 20:01:20.166680 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:20.166655    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"
Sep 11 20:01:20.293862 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:20.293783    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1cbb2-1878-4aa0-a65d-e07c7bcc3902-maas-api-service-ca\") pod \"56e1cbb2-1878-4aa0-a65d-e07c7bcc3902\" (UID: \"56e1cbb2-1878-4aa0-a65d-e07c7bcc3902\") "
Sep 11 20:01:20.294156 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:20.294133    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e1cbb2-1878-4aa0-a65d-e07c7bcc3902-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56e1cbb2-1878-4aa0-a65d-e07c7bcc3902" (UID: "56e1cbb2-1878-4aa0-a65d-e07c7bcc3902"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:20.395322 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:20.395293    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56e1cbb2-1878-4aa0-a65d-e07c7bcc3902-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:21.046300 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:21.046269    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f" event={"ID":"56e1cbb2-1878-4aa0-a65d-e07c7bcc3902","Type":"ContainerDied","Data":"b21db5ad5435f1110a6783056eba12db710132fa2f5524621c0cc2c8863bb244"}
Sep 11 20:01:21.046300 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:21.046291    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"
Sep 11 20:01:21.046519 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:21.046317    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b21db5ad5435f1110a6783056eba12db710132fa2f5524621c0cc2c8863bb244"
Sep 11 20:01:22.003302 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:22.003264    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:01:22.238490 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:22.238463    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"]
Sep 11 20:01:22.244802 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:22.244777    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"]
Sep 11 20:01:22.258260 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:22.258194    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"]
Sep 11 20:01:25.334523 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.334491    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"]
Sep 11 20:01:25.334933 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.334721    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw" podUID="06186a8c-cae4-47cd-89c6-23b4f50a4a30" containerName="payload-processing" containerID="cri-o://fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87" gracePeriod=30
Sep 11 20:01:25.346355 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.346270    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"]
Sep 11 20:01:25.346555 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.346505    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s" podUID="a7d5a7dd-ce18-4a20-b936-c8d02a7245cd" containerName="payload-pre-processing" containerID="cri-o://77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072" gracePeriod=30
Sep 11 20:01:25.496358 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.496325    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"]
Sep 11 20:01:25.500818 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.500792    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1a074-qtpxn"]
Sep 11 20:01:25.625422 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.625397    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:01:25.628180 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.628163    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:01:25.741289 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.741260    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06186a8c-cae4-47cd-89c6-23b4f50a4a30-plugins-config-volume\") pod \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\" (UID: \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\") "
Sep 11 20:01:25.741453 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.741330    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glz8h\" (UniqueName: \"kubernetes.io/projected/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-kube-api-access-glz8h\") pod \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\" (UID: \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\") "
Sep 11 20:01:25.741453 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.741359    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-plugins-config-volume\") pod \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\" (UID: \"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd\") "
Sep 11 20:01:25.741453 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.741399    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksj9x\" (UniqueName: \"kubernetes.io/projected/06186a8c-cae4-47cd-89c6-23b4f50a4a30-kube-api-access-ksj9x\") pod \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\" (UID: \"06186a8c-cae4-47cd-89c6-23b4f50a4a30\") "
Sep 11 20:01:25.741724 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.741698    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06186a8c-cae4-47cd-89c6-23b4f50a4a30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06186a8c-cae4-47cd-89c6-23b4f50a4a30" (UID: "06186a8c-cae4-47cd-89c6-23b4f50a4a30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:25.741794 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.741732    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7d5a7dd-ce18-4a20-b936-c8d02a7245cd" (UID: "a7d5a7dd-ce18-4a20-b936-c8d02a7245cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:25.743503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.743473    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-kube-api-access-glz8h" (OuterVolumeSpecName: "kube-api-access-glz8h") pod "a7d5a7dd-ce18-4a20-b936-c8d02a7245cd" (UID: "a7d5a7dd-ce18-4a20-b936-c8d02a7245cd"). InnerVolumeSpecName "kube-api-access-glz8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:25.743612 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.743477    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06186a8c-cae4-47cd-89c6-23b4f50a4a30-kube-api-access-ksj9x" (OuterVolumeSpecName: "kube-api-access-ksj9x") pod "06186a8c-cae4-47cd-89c6-23b4f50a4a30" (UID: "06186a8c-cae4-47cd-89c6-23b4f50a4a30"). InnerVolumeSpecName "kube-api-access-ksj9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:25.842592 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.842566    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:25.842592 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.842587    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksj9x\" (UniqueName: \"kubernetes.io/projected/06186a8c-cae4-47cd-89c6-23b4f50a4a30-kube-api-access-ksj9x\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:25.842592 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.842597    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06186a8c-cae4-47cd-89c6-23b4f50a4a30-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:25.842769 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.842605    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glz8h\" (UniqueName: \"kubernetes.io/projected/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd-kube-api-access-glz8h\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:25.874587 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:25.874562    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863f583f-2242-439b-aaf7-cacd98ed98a6" path="/var/lib/kubelet/pods/863f583f-2242-439b-aaf7-cacd98ed98a6/volumes"
Sep 11 20:01:26.074179 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.074145    2571 generic.go:356] "Generic (PLEG): container finished" podID="a7d5a7dd-ce18-4a20-b936-c8d02a7245cd" containerID="77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072" exitCode=0
Sep 11 20:01:26.074333 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.074183    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"
Sep 11 20:01:26.074333 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.074239    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s" event={"ID":"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd","Type":"ContainerDied","Data":"77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072"}
Sep 11 20:01:26.074333 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.074293    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s" event={"ID":"a7d5a7dd-ce18-4a20-b936-c8d02a7245cd","Type":"ContainerDied","Data":"70f406a19cbcf8cf83d78606d4667a8a92fe05783ce510b94b8af73897678242"}
Sep 11 20:01:26.074333 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.074320    2571 scope.go:117] "RemoveContainer" containerID="77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072"
Sep 11 20:01:26.075568 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.075549    2571 generic.go:356] "Generic (PLEG): container finished" podID="06186a8c-cae4-47cd-89c6-23b4f50a4a30" containerID="fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87" exitCode=0
Sep 11 20:01:26.075669 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.075583    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw" event={"ID":"06186a8c-cae4-47cd-89c6-23b4f50a4a30","Type":"ContainerDied","Data":"fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87"}
Sep 11 20:01:26.075669 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.075602    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"
Sep 11 20:01:26.075669 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.075638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw" event={"ID":"06186a8c-cae4-47cd-89c6-23b4f50a4a30","Type":"ContainerDied","Data":"4a10fde066819d52f9d90c79bbaa950e885de4589cdf393bbeb4b8550167115b"}
Sep 11 20:01:26.083382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.083369    2571 scope.go:117] "RemoveContainer" containerID="77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072"
Sep 11 20:01:26.083700 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:26.083677    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072\": container with ID starting with 77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072 not found: ID does not exist" containerID="77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072"
Sep 11 20:01:26.083801 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.083704    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072"} err="failed to get container status \"77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072\": rpc error: code = NotFound desc = could not find container \"77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072\": container with ID starting with 77d28c1d3d9065e64a0f9fe8f5b5aab74ef4320f5177153c4662a698c651e072 not found: ID does not exist"
Sep 11 20:01:26.083801 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.083719    2571 scope.go:117] "RemoveContainer" containerID="fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87"
Sep 11 20:01:26.091557 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.091528    2571 scope.go:117] "RemoveContainer" containerID="fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87"
Sep 11 20:01:26.091939 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:26.091916    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87\": container with ID starting with fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87 not found: ID does not exist" containerID="fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87"
Sep 11 20:01:26.092029 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.091948    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87"} err="failed to get container status \"fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87\": rpc error: code = NotFound desc = could not find container \"fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87\": container with ID starting with fce6c63e8900435db18afac122c26cf6805e92f3e7540f56acdba50db9d18b87 not found: ID does not exist"
Sep 11 20:01:26.094641 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.094622    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"]
Sep 11 20:01:26.098611 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.098591    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1a074-684b9567c4-2xf8s"]
Sep 11 20:01:26.109882 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.109860    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"]
Sep 11 20:01:26.116532 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:26.116506    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1a074-6bb9b75cc9-jbblw"]
Sep 11 20:01:27.873409 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:27.873372    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06186a8c-cae4-47cd-89c6-23b4f50a4a30" path="/var/lib/kubelet/pods/06186a8c-cae4-47cd-89c6-23b4f50a4a30/volumes"
Sep 11 20:01:27.873859 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:27.873739    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d5a7dd-ce18-4a20-b936-c8d02a7245cd" path="/var/lib/kubelet/pods/a7d5a7dd-ce18-4a20-b936-c8d02a7245cd/volumes"
Sep 11 20:01:32.221673 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.221637    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"]
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.221944    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06186a8c-cae4-47cd-89c6-23b4f50a4a30" containerName="payload-processing"
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.221956    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="06186a8c-cae4-47cd-89c6-23b4f50a4a30" containerName="payload-processing"
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.221970    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e1cbb2-1878-4aa0-a65d-e07c7bcc3902" containerName="revoke-keys"
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.221976    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e1cbb2-1878-4aa0-a65d-e07c7bcc3902" containerName="revoke-keys"
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.221985    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d5a7dd-ce18-4a20-b936-c8d02a7245cd" containerName="payload-pre-processing"
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.221990    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d5a7dd-ce18-4a20-b936-c8d02a7245cd" containerName="payload-pre-processing"
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.222038    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd493de-1d79-427c-aa36-b2f2725a4a44" containerName="istio-proxy"
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.222046    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="06186a8c-cae4-47cd-89c6-23b4f50a4a30" containerName="payload-processing"
Sep 11 20:01:32.222049 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.222052    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d5a7dd-ce18-4a20-b936-c8d02a7245cd" containerName="payload-pre-processing"
Sep 11 20:01:32.222369 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.222058    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e1cbb2-1878-4aa0-a65d-e07c7bcc3902" containerName="revoke-keys"
Sep 11 20:01:32.225168 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.225152    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"
Sep 11 20:01:32.228221 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.228203    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:01:32.228335 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.228314    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:01:32.236061 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.236027    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"]
Sep 11 20:01:32.298206 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.298177    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610c3c17-4d2d-46b7-b928-44f5d4e3674a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x\" (UID: \"610c3c17-4d2d-46b7-b928-44f5d4e3674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"
Sep 11 20:01:32.399682 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.399642    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610c3c17-4d2d-46b7-b928-44f5d4e3674a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x\" (UID: \"610c3c17-4d2d-46b7-b928-44f5d4e3674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"
Sep 11 20:01:32.400302 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.400281    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610c3c17-4d2d-46b7-b928-44f5d4e3674a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x\" (UID: \"610c3c17-4d2d-46b7-b928-44f5d4e3674a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"
Sep 11 20:01:32.535486 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.535403    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"
Sep 11 20:01:32.808645 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:32.808611    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"]
Sep 11 20:01:32.810044 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:01:32.810016    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610c3c17_4d2d_46b7_b928_44f5d4e3674a.slice/crio-b29874e2d6830ca512fa8834bd494208adb75957143e74c1ba5bc7ef1f782a54 WatchSource:0}: Error finding container b29874e2d6830ca512fa8834bd494208adb75957143e74c1ba5bc7ef1f782a54: Status 404 returned error can't find the container with id b29874e2d6830ca512fa8834bd494208adb75957143e74c1ba5bc7ef1f782a54
Sep 11 20:01:33.123686 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:33.123661    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x" event={"ID":"610c3c17-4d2d-46b7-b928-44f5d4e3674a","Type":"ContainerStarted","Data":"949a61888b276621a9b0d7cc59d929b90be25dfca21910868992c989e8957b30"}
Sep 11 20:01:33.123798 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:33.123710    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x" event={"ID":"610c3c17-4d2d-46b7-b928-44f5d4e3674a","Type":"ContainerStarted","Data":"b29874e2d6830ca512fa8834bd494208adb75957143e74c1ba5bc7ef1f782a54"}
Sep 11 20:01:33.143051 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:33.142889    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x" podStartSLOduration=1.142870993 podStartE2EDuration="1.142870993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:33.142005704 +0000 UTC m=+1929.817942612" watchObservedRunningTime="2026-09-11 20:01:33.142870993 +0000 UTC m=+1929.818807907"
Sep 11 20:01:34.028975 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:34.028937    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"]
Sep 11 20:01:34.031047 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:34.031026    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a8ee95-p657f"]
Sep 11 20:01:34.131598 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:34.131568    2571 generic.go:356] "Generic (PLEG): container finished" podID="610c3c17-4d2d-46b7-b928-44f5d4e3674a" containerID="949a61888b276621a9b0d7cc59d929b90be25dfca21910868992c989e8957b30" exitCode=0
Sep 11 20:01:34.131728 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:34.131651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x" event={"ID":"610c3c17-4d2d-46b7-b928-44f5d4e3674a","Type":"ContainerDied","Data":"949a61888b276621a9b0d7cc59d929b90be25dfca21910868992c989e8957b30"}
Sep 11 20:01:35.259083 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:35.259061    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"
Sep 11 20:01:35.326324 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:35.326295    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610c3c17-4d2d-46b7-b928-44f5d4e3674a-maas-api-service-ca\") pod \"610c3c17-4d2d-46b7-b928-44f5d4e3674a\" (UID: \"610c3c17-4d2d-46b7-b928-44f5d4e3674a\") "
Sep 11 20:01:35.326681 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:35.326661    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610c3c17-4d2d-46b7-b928-44f5d4e3674a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "610c3c17-4d2d-46b7-b928-44f5d4e3674a" (UID: "610c3c17-4d2d-46b7-b928-44f5d4e3674a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:35.427377 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:35.427320    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/610c3c17-4d2d-46b7-b928-44f5d4e3674a-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:35.872782 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:35.872753    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e1cbb2-1878-4aa0-a65d-e07c7bcc3902" path="/var/lib/kubelet/pods/56e1cbb2-1878-4aa0-a65d-e07c7bcc3902/volumes"
Sep 11 20:01:36.144982 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:36.144898    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"
Sep 11 20:01:36.144982 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:36.144942    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x" event={"ID":"610c3c17-4d2d-46b7-b928-44f5d4e3674a","Type":"ContainerDied","Data":"b29874e2d6830ca512fa8834bd494208adb75957143e74c1ba5bc7ef1f782a54"}
Sep 11 20:01:36.145186 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:36.144990    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b29874e2d6830ca512fa8834bd494208adb75957143e74c1ba5bc7ef1f782a54"
Sep 11 20:01:37.257655 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.257594    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44" podUID="2993a257-1071-4519-bc3b-4ae5f9216ebd" containerName="main" containerID="cri-o://65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec" gracePeriod=45
Sep 11 20:01:37.263298 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.263271    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx" podUID="d20a978b-7298-46f0-aece-b8cd25938cf2" containerName="main" containerID="cri-o://35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a" gracePeriod=45
Sep 11 20:01:37.276560 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.276527    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq" podUID="c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" containerName="main" containerID="cri-o://c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6" gracePeriod=45
Sep 11 20:01:37.527073 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.527049    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 20:01:37.552402 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.552261    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 20:01:37.559954 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.559931    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 20:01:37.647784 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.647755    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-dshm\") pod \"d20a978b-7298-46f0-aece-b8cd25938cf2\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") "
Sep 11 20:01:37.647784 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.647787    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-dshm\") pod \"2993a257-1071-4519-bc3b-4ae5f9216ebd\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") "
Sep 11 20:01:37.648030 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.647810    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-model-cache\") pod \"d20a978b-7298-46f0-aece-b8cd25938cf2\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") "
Sep 11 20:01:37.648030 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.647829    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdh65\" (UniqueName: \"kubernetes.io/projected/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-kube-api-access-rdh65\") pod \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") "
Sep 11 20:01:37.648030 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.647848    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-model-cache\") pod \"2993a257-1071-4519-bc3b-4ae5f9216ebd\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") "
Sep 11 20:01:37.648030 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.647874    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20a978b-7298-46f0-aece-b8cd25938cf2-tls-certs\") pod \"d20a978b-7298-46f0-aece-b8cd25938cf2\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") "
Sep 11 20:01:37.648030 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.647922    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-tmp-dir\") pod \"2993a257-1071-4519-bc3b-4ae5f9216ebd\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") "
Sep 11 20:01:37.648030 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.647963    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-tmp-dir\") pod \"d20a978b-7298-46f0-aece-b8cd25938cf2\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") "
Sep 11 20:01:37.648030 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648012    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-home\") pod \"2993a257-1071-4519-bc3b-4ae5f9216ebd\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") "
Sep 11 20:01:37.648428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648062    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-home\") pod \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") "
Sep 11 20:01:37.648428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648094    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993a257-1071-4519-bc3b-4ae5f9216ebd-tls-certs\") pod \"2993a257-1071-4519-bc3b-4ae5f9216ebd\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") "
Sep 11 20:01:37.648428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648152    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tmp-dir\") pod \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") "
Sep 11 20:01:37.648428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648188    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsdbj\" (UniqueName: \"kubernetes.io/projected/d20a978b-7298-46f0-aece-b8cd25938cf2-kube-api-access-zsdbj\") pod \"d20a978b-7298-46f0-aece-b8cd25938cf2\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") "
Sep 11 20:01:37.648428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648227    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4ns2\" (UniqueName: \"kubernetes.io/projected/2993a257-1071-4519-bc3b-4ae5f9216ebd-kube-api-access-w4ns2\") pod \"2993a257-1071-4519-bc3b-4ae5f9216ebd\" (UID: \"2993a257-1071-4519-bc3b-4ae5f9216ebd\") "
Sep 11 20:01:37.648428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648260    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-dshm\") pod \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") "
Sep 11 20:01:37.648428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648283    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-home\") pod \"d20a978b-7298-46f0-aece-b8cd25938cf2\" (UID: \"d20a978b-7298-46f0-aece-b8cd25938cf2\") "
Sep 11 20:01:37.648428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648325    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tls-certs\") pod \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") "
Sep 11 20:01:37.648852 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648084    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-model-cache" (OuterVolumeSpecName: "model-cache") pod "d20a978b-7298-46f0-aece-b8cd25938cf2" (UID: "d20a978b-7298-46f0-aece-b8cd25938cf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.648852 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648556    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-home" (OuterVolumeSpecName: "home") pod "2993a257-1071-4519-bc3b-4ae5f9216ebd" (UID: "2993a257-1071-4519-bc3b-4ae5f9216ebd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.648852 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648295    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d20a978b-7298-46f0-aece-b8cd25938cf2" (UID: "d20a978b-7298-46f0-aece-b8cd25938cf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.648852 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648304    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-home" (OuterVolumeSpecName: "home") pod "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" (UID: "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.648852 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648495    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2993a257-1071-4519-bc3b-4ae5f9216ebd" (UID: "2993a257-1071-4519-bc3b-4ae5f9216ebd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.648852 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.648536    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-model-cache" (OuterVolumeSpecName: "model-cache") pod "2993a257-1071-4519-bc3b-4ae5f9216ebd" (UID: "2993a257-1071-4519-bc3b-4ae5f9216ebd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649208    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-home" (OuterVolumeSpecName: "home") pod "d20a978b-7298-46f0-aece-b8cd25938cf2" (UID: "d20a978b-7298-46f0-aece-b8cd25938cf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649392    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-model-cache\") pod \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\" (UID: \"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86\") "
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649403    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" (UID: "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649713    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649733    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649746    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649758    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649770    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649782    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649794    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.649808    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.650306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.650018    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" (UID: "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.651237 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.650761    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-dshm" (OuterVolumeSpecName: "dshm") pod "2993a257-1071-4519-bc3b-4ae5f9216ebd" (UID: "2993a257-1071-4519-bc3b-4ae5f9216ebd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.651237 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.650806    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-kube-api-access-rdh65" (OuterVolumeSpecName: "kube-api-access-rdh65") pod "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" (UID: "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86"). InnerVolumeSpecName "kube-api-access-rdh65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:37.651476 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.651448    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" (UID: "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:01:37.651555 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.651480    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2993a257-1071-4519-bc3b-4ae5f9216ebd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2993a257-1071-4519-bc3b-4ae5f9216ebd" (UID: "2993a257-1071-4519-bc3b-4ae5f9216ebd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:01:37.651555 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.651484    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-dshm" (OuterVolumeSpecName: "dshm") pod "d20a978b-7298-46f0-aece-b8cd25938cf2" (UID: "d20a978b-7298-46f0-aece-b8cd25938cf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.651555 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.651475    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-dshm" (OuterVolumeSpecName: "dshm") pod "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" (UID: "c7b8a9a5-78ec-4f02-8da7-ef4465bbce86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:37.651798 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.651783    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20a978b-7298-46f0-aece-b8cd25938cf2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d20a978b-7298-46f0-aece-b8cd25938cf2" (UID: "d20a978b-7298-46f0-aece-b8cd25938cf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:01:37.651967 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.651948    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2993a257-1071-4519-bc3b-4ae5f9216ebd-kube-api-access-w4ns2" (OuterVolumeSpecName: "kube-api-access-w4ns2") pod "2993a257-1071-4519-bc3b-4ae5f9216ebd" (UID: "2993a257-1071-4519-bc3b-4ae5f9216ebd"). InnerVolumeSpecName "kube-api-access-w4ns2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:37.652153 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.652132    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20a978b-7298-46f0-aece-b8cd25938cf2-kube-api-access-zsdbj" (OuterVolumeSpecName: "kube-api-access-zsdbj") pod "d20a978b-7298-46f0-aece-b8cd25938cf2" (UID: "d20a978b-7298-46f0-aece-b8cd25938cf2"). InnerVolumeSpecName "kube-api-access-zsdbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:37.750704 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750659    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20a978b-7298-46f0-aece-b8cd25938cf2-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750704 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750698    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993a257-1071-4519-bc3b-4ae5f9216ebd-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750704 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750707    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdbj\" (UniqueName: \"kubernetes.io/projected/d20a978b-7298-46f0-aece-b8cd25938cf2-kube-api-access-zsdbj\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750704 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750718    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ns2\" (UniqueName: \"kubernetes.io/projected/2993a257-1071-4519-bc3b-4ae5f9216ebd-kube-api-access-w4ns2\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750957 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750727    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750957 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750735    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750957 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750743    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750957 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750751    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20a978b-7298-46f0-aece-b8cd25938cf2-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750957 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750757    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993a257-1071-4519-bc3b-4ae5f9216ebd-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:37.750957 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:37.750765    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdh65\" (UniqueName: \"kubernetes.io/projected/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86-kube-api-access-rdh65\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:38.158324 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.158240    2571 generic.go:356] "Generic (PLEG): container finished" podID="c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" containerID="c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6" exitCode=0
Sep 11 20:01:38.158324 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.158296    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"
Sep 11 20:01:38.158532 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.158324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq" event={"ID":"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86","Type":"ContainerDied","Data":"c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6"}
Sep 11 20:01:38.158532 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.158364    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq" event={"ID":"c7b8a9a5-78ec-4f02-8da7-ef4465bbce86","Type":"ContainerDied","Data":"0d0b90b4b7dffff27b6b43a40eb847f09cfb09337b5572c51b07d517517503c4"}
Sep 11 20:01:38.158532 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.158382    2571 scope.go:117] "RemoveContainer" containerID="c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6"
Sep 11 20:01:38.159943 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.159922    2571 generic.go:356] "Generic (PLEG): container finished" podID="2993a257-1071-4519-bc3b-4ae5f9216ebd" containerID="65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec" exitCode=0
Sep 11 20:01:38.160085 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.159987    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44" event={"ID":"2993a257-1071-4519-bc3b-4ae5f9216ebd","Type":"ContainerDied","Data":"65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec"}
Sep 11 20:01:38.160085 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.160004    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"
Sep 11 20:01:38.160085 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.160025    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44" event={"ID":"2993a257-1071-4519-bc3b-4ae5f9216ebd","Type":"ContainerDied","Data":"46c6c74eb5cefbbbd0e8f4ae6d73331efc549c9890b0c0868f518f7043755d73"}
Sep 11 20:01:38.161743 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.161715    2571 generic.go:356] "Generic (PLEG): container finished" podID="d20a978b-7298-46f0-aece-b8cd25938cf2" containerID="35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a" exitCode=0
Sep 11 20:01:38.161880 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.161748    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx" event={"ID":"d20a978b-7298-46f0-aece-b8cd25938cf2","Type":"ContainerDied","Data":"35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a"}
Sep 11 20:01:38.161880 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.161772    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx" event={"ID":"d20a978b-7298-46f0-aece-b8cd25938cf2","Type":"ContainerDied","Data":"a756b0cf9cec9eb630a3d9e3a22475f1761ab3616531cce818e42756e10f04c9"}
Sep 11 20:01:38.161880 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.161790    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"
Sep 11 20:01:38.168640 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.168616    2571 scope.go:117] "RemoveContainer" containerID="c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6"
Sep 11 20:01:38.168904 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:38.168884    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6\": container with ID starting with c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6 not found: ID does not exist" containerID="c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6"
Sep 11 20:01:38.168958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.168911    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6"} err="failed to get container status \"c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6\": rpc error: code = NotFound desc = could not find container \"c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6\": container with ID starting with c7cec3ed1b1319c2dd0c7638f7494f3a849f5f829588dd67419325bbeaa95fa6 not found: ID does not exist"
Sep 11 20:01:38.168958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.168927    2571 scope.go:117] "RemoveContainer" containerID="65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec"
Sep 11 20:01:38.176804 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.176786    2571 scope.go:117] "RemoveContainer" containerID="65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec"
Sep 11 20:01:38.177082 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:38.177066    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec\": container with ID starting with 65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec not found: ID does not exist" containerID="65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec"
Sep 11 20:01:38.177186 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.177090    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec"} err="failed to get container status \"65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec\": rpc error: code = NotFound desc = could not find container \"65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec\": container with ID starting with 65b71e1f5b2e6a35c303d17bfd302f460a67496452224797984be23376f624ec not found: ID does not exist"
Sep 11 20:01:38.177186 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.177129    2571 scope.go:117] "RemoveContainer" containerID="35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a"
Sep 11 20:01:38.182061 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.182037    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"]
Sep 11 20:01:38.184947 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.184934    2571 scope.go:117] "RemoveContainer" containerID="35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a"
Sep 11 20:01:38.185227 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:38.185202    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a\": container with ID starting with 35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a not found: ID does not exist" containerID="35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a"
Sep 11 20:01:38.185300 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.185234    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a"} err="failed to get container status \"35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a\": rpc error: code = NotFound desc = could not find container \"35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a\": container with ID starting with 35b8f1b71a47434cc6d4ed1912f9eeaf3574e00c3ca0035eae3192979897375a not found: ID does not exist"
Sep 11 20:01:38.188112 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.188079    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/distinct-w4-a8ee95-kserve-b548f7498-mcv44"]
Sep 11 20:01:38.198305 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.198282    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"]
Sep 11 20:01:38.202626 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.202605    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/premium-w4-a8ee95-kserve-75d86f96b7-2pvrx"]
Sep 11 20:01:38.214613 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.214592    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"]
Sep 11 20:01:38.218553 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:38.218533    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a8ee95/simulator-w4-a8ee95-kserve-75f7cc79c4-vrwnq"]
Sep 11 20:01:39.872540 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:39.872505    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2993a257-1071-4519-bc3b-4ae5f9216ebd" path="/var/lib/kubelet/pods/2993a257-1071-4519-bc3b-4ae5f9216ebd/volumes"
Sep 11 20:01:39.872900 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:39.872878    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" path="/var/lib/kubelet/pods/c7b8a9a5-78ec-4f02-8da7-ef4465bbce86/volumes"
Sep 11 20:01:39.873238 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:39.873225    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20a978b-7298-46f0-aece-b8cd25938cf2" path="/var/lib/kubelet/pods/d20a978b-7298-46f0-aece-b8cd25938cf2/volumes"
Sep 11 20:01:43.090416 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.090377    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"]
Sep 11 20:01:43.093256 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.093227    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e8cfcf-29819280-6lwhs"]
Sep 11 20:01:43.124632 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.124603    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"]
Sep 11 20:01:43.124902 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.124868    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv" podUID="bd29eeb3-6957-4a8e-80f8-14d42ce6eeac" containerName="payload-processing" containerID="cri-o://2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79" gracePeriod=30
Sep 11 20:01:43.304806 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.304774    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"]
Sep 11 20:01:43.310920 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.310892    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e8cfcf-56g5x"]
Sep 11 20:01:43.360913 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.360886    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 20:01:43.498332 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.498300    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-plugins-config-volume\") pod \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\" (UID: \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\") "
Sep 11 20:01:43.498489 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.498364    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k7bc\" (UniqueName: \"kubernetes.io/projected/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-kube-api-access-2k7bc\") pod \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\" (UID: \"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac\") "
Sep 11 20:01:43.498653 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.498627    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd29eeb3-6957-4a8e-80f8-14d42ce6eeac" (UID: "bd29eeb3-6957-4a8e-80f8-14d42ce6eeac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:43.500383 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.500357    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-kube-api-access-2k7bc" (OuterVolumeSpecName: "kube-api-access-2k7bc") pod "bd29eeb3-6957-4a8e-80f8-14d42ce6eeac" (UID: "bd29eeb3-6957-4a8e-80f8-14d42ce6eeac"). InnerVolumeSpecName "kube-api-access-2k7bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:43.599528 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.599445    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k7bc\" (UniqueName: \"kubernetes.io/projected/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-kube-api-access-2k7bc\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:43.599528 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.599475    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:43.874670 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.874629    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610c3c17-4d2d-46b7-b928-44f5d4e3674a" path="/var/lib/kubelet/pods/610c3c17-4d2d-46b7-b928-44f5d4e3674a/volumes"
Sep 11 20:01:43.875028 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:43.875005    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2" path="/var/lib/kubelet/pods/74273c4b-9aeb-4f0e-a7d6-3f0fc71176d2/volumes"
Sep 11 20:01:44.198214 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.198130    2571 generic.go:356] "Generic (PLEG): container finished" podID="bd29eeb3-6957-4a8e-80f8-14d42ce6eeac" containerID="2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79" exitCode=0
Sep 11 20:01:44.198602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.198220    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"
Sep 11 20:01:44.198602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.198262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv" event={"ID":"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac","Type":"ContainerDied","Data":"2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79"}
Sep 11 20:01:44.198602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.198317    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv" event={"ID":"bd29eeb3-6957-4a8e-80f8-14d42ce6eeac","Type":"ContainerDied","Data":"36dedb18d84937f07c699a56adf21e6fad8ee3e66c6a8f226fd0c241f905a247"}
Sep 11 20:01:44.198602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.198338    2571 scope.go:117] "RemoveContainer" containerID="2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79"
Sep 11 20:01:44.207170 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.207152    2571 scope.go:117] "RemoveContainer" containerID="2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79"
Sep 11 20:01:44.207450 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:01:44.207428    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79\": container with ID starting with 2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79 not found: ID does not exist" containerID="2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79"
Sep 11 20:01:44.207549 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.207459    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79"} err="failed to get container status \"2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79\": rpc error: code = NotFound desc = could not find container \"2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79\": container with ID starting with 2560a2d3d5318fcef3da0e4d295eefd35cd0e16ef0a92609c99a0264f40cea79 not found: ID does not exist"
Sep 11 20:01:44.220916 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.220884    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"]
Sep 11 20:01:44.223797 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:44.223777    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8cfcf-5c669f8cf5-khqdv"]
Sep 11 20:01:45.458488 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:45.458452    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:01:45.873709 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:45.873679    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd29eeb3-6957-4a8e-80f8-14d42ce6eeac" path="/var/lib/kubelet/pods/bd29eeb3-6957-4a8e-80f8-14d42ce6eeac/volumes"
Sep 11 20:01:49.990985 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.990942    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"]
Sep 11 20:01:49.991464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991366    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2993a257-1071-4519-bc3b-4ae5f9216ebd" containerName="main"
Sep 11 20:01:49.991464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991384    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2993a257-1071-4519-bc3b-4ae5f9216ebd" containerName="main"
Sep 11 20:01:49.991464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991395    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd29eeb3-6957-4a8e-80f8-14d42ce6eeac" containerName="payload-processing"
Sep 11 20:01:49.991464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991404    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd29eeb3-6957-4a8e-80f8-14d42ce6eeac" containerName="payload-processing"
Sep 11 20:01:49.991464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991416    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20a978b-7298-46f0-aece-b8cd25938cf2" containerName="main"
Sep 11 20:01:49.991464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991425    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20a978b-7298-46f0-aece-b8cd25938cf2" containerName="main"
Sep 11 20:01:49.991464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991436    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610c3c17-4d2d-46b7-b928-44f5d4e3674a" containerName="revoke-keys"
Sep 11 20:01:49.991464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991443    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="610c3c17-4d2d-46b7-b928-44f5d4e3674a" containerName="revoke-keys"
Sep 11 20:01:49.991854 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991469    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" containerName="main"
Sep 11 20:01:49.991854 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991477    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" containerName="main"
Sep 11 20:01:49.991854 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991556    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="610c3c17-4d2d-46b7-b928-44f5d4e3674a" containerName="revoke-keys"
Sep 11 20:01:49.991854 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991569    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd29eeb3-6957-4a8e-80f8-14d42ce6eeac" containerName="payload-processing"
Sep 11 20:01:49.991854 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991580    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2993a257-1071-4519-bc3b-4ae5f9216ebd" containerName="main"
Sep 11 20:01:49.991854 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991591    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b8a9a5-78ec-4f02-8da7-ef4465bbce86" containerName="main"
Sep 11 20:01:49.991854 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.991600    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20a978b-7298-46f0-aece-b8cd25938cf2" containerName="main"
Sep 11 20:01:49.995838 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.995818    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"
Sep 11 20:01:49.998340 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.998316    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:01:49.998582 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:49.998561    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:01:50.002771 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:50.002740    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"]
Sep 11 20:01:50.149062 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:50.149014    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c0804-cfe4-4690-9d9a-3bcbca306bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs\" (UID: \"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"
Sep 11 20:01:50.249955 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:50.249869    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c0804-cfe4-4690-9d9a-3bcbca306bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs\" (UID: \"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"
Sep 11 20:01:50.250529 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:50.250502    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c0804-cfe4-4690-9d9a-3bcbca306bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs\" (UID: \"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"
Sep 11 20:01:50.307740 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:50.307702    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"
Sep 11 20:01:50.576837 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:50.576813    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"]
Sep 11 20:01:50.578585 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:01:50.578557    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6c0804_cfe4_4690_9d9a_3bcbca306bdd.slice/crio-5b78949b996dba845fd65004cc4ceed2adcf3103dd70a4adf047189af0d251c8 WatchSource:0}: Error finding container 5b78949b996dba845fd65004cc4ceed2adcf3103dd70a4adf047189af0d251c8: Status 404 returned error can't find the container with id 5b78949b996dba845fd65004cc4ceed2adcf3103dd70a4adf047189af0d251c8
Sep 11 20:01:51.241988 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:51.241956    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs" event={"ID":"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd","Type":"ContainerStarted","Data":"eff199182b4abf4bbcd3311ddc9280e721003cd8997b48c3ec4de1af294a1b01"}
Sep 11 20:01:51.242361 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:51.242012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs" event={"ID":"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd","Type":"ContainerStarted","Data":"5b78949b996dba845fd65004cc4ceed2adcf3103dd70a4adf047189af0d251c8"}
Sep 11 20:01:51.258917 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:51.258872    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs" podStartSLOduration=2.258855184 podStartE2EDuration="2.258855184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:51.257508702 +0000 UTC m=+1947.933445612" watchObservedRunningTime="2026-09-11 20:01:51.258855184 +0000 UTC m=+1947.934792091"
Sep 11 20:01:52.248549 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:52.248513    2571 generic.go:356] "Generic (PLEG): container finished" podID="6d6c0804-cfe4-4690-9d9a-3bcbca306bdd" containerID="eff199182b4abf4bbcd3311ddc9280e721003cd8997b48c3ec4de1af294a1b01" exitCode=0
Sep 11 20:01:52.248924 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:52.248575    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs" event={"ID":"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd","Type":"ContainerDied","Data":"eff199182b4abf4bbcd3311ddc9280e721003cd8997b48c3ec4de1af294a1b01"}
Sep 11 20:01:53.380657 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:53.380633    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"
Sep 11 20:01:53.473691 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:53.473660    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c0804-cfe4-4690-9d9a-3bcbca306bdd-maas-api-service-ca\") pod \"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd\" (UID: \"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd\") "
Sep 11 20:01:53.473989 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:53.473966    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6c0804-cfe4-4690-9d9a-3bcbca306bdd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d6c0804-cfe4-4690-9d9a-3bcbca306bdd" (UID: "6d6c0804-cfe4-4690-9d9a-3bcbca306bdd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:53.574331 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:53.574264    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6c0804-cfe4-4690-9d9a-3bcbca306bdd-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:01:54.260838 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:54.260752    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs" event={"ID":"6d6c0804-cfe4-4690-9d9a-3bcbca306bdd","Type":"ContainerDied","Data":"5b78949b996dba845fd65004cc4ceed2adcf3103dd70a4adf047189af0d251c8"}
Sep 11 20:01:54.260838 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:54.260775    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"
Sep 11 20:01:54.260838 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:01:54.260811    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b78949b996dba845fd65004cc4ceed2adcf3103dd70a4adf047189af0d251c8"
Sep 11 20:02:00.520577 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.520536    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"]
Sep 11 20:02:00.521536 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.521480    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp" podUID="f705fabd-4ccc-4926-af16-43cdef7d4f27" containerName="payload-pre-processing" containerID="cri-o://f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b" gracePeriod=30
Sep 11 20:02:00.672864 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.672834    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"]
Sep 11 20:02:00.674954 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.674928    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f7fd24-hptzs"]
Sep 11 20:02:00.766486 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.766463    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 20:02:00.818989 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.818928    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bsh4\" (UniqueName: \"kubernetes.io/projected/f705fabd-4ccc-4926-af16-43cdef7d4f27-kube-api-access-9bsh4\") pod \"f705fabd-4ccc-4926-af16-43cdef7d4f27\" (UID: \"f705fabd-4ccc-4926-af16-43cdef7d4f27\") "
Sep 11 20:02:00.819120 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.818995    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f705fabd-4ccc-4926-af16-43cdef7d4f27-plugins-config-volume\") pod \"f705fabd-4ccc-4926-af16-43cdef7d4f27\" (UID: \"f705fabd-4ccc-4926-af16-43cdef7d4f27\") "
Sep 11 20:02:00.819354 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.819332    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f705fabd-4ccc-4926-af16-43cdef7d4f27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f705fabd-4ccc-4926-af16-43cdef7d4f27" (UID: "f705fabd-4ccc-4926-af16-43cdef7d4f27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:02:00.820953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.820935    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f705fabd-4ccc-4926-af16-43cdef7d4f27-kube-api-access-9bsh4" (OuterVolumeSpecName: "kube-api-access-9bsh4") pod "f705fabd-4ccc-4926-af16-43cdef7d4f27" (UID: "f705fabd-4ccc-4926-af16-43cdef7d4f27"). InnerVolumeSpecName "kube-api-access-9bsh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:02:00.920280 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.920252    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f705fabd-4ccc-4926-af16-43cdef7d4f27-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:00.920280 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:00.920274    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bsh4\" (UniqueName: \"kubernetes.io/projected/f705fabd-4ccc-4926-af16-43cdef7d4f27-kube-api-access-9bsh4\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:01.298487 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.298454    2571 generic.go:356] "Generic (PLEG): container finished" podID="f705fabd-4ccc-4926-af16-43cdef7d4f27" containerID="f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b" exitCode=0
Sep 11 20:02:01.298653 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.298502    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"
Sep 11 20:02:01.298653 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.298517    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp" event={"ID":"f705fabd-4ccc-4926-af16-43cdef7d4f27","Type":"ContainerDied","Data":"f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b"}
Sep 11 20:02:01.298653 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.298557    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp" event={"ID":"f705fabd-4ccc-4926-af16-43cdef7d4f27","Type":"ContainerDied","Data":"b3213c7f17ae13975f57542fddedc0e578222d224c8412b1460686cbf440adbd"}
Sep 11 20:02:01.298653 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.298579    2571 scope.go:117] "RemoveContainer" containerID="f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b"
Sep 11 20:02:01.310690 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.310668    2571 scope.go:117] "RemoveContainer" containerID="f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b"
Sep 11 20:02:01.311077 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:02:01.311058    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b\": container with ID starting with f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b not found: ID does not exist" containerID="f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b"
Sep 11 20:02:01.311193 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.311087    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b"} err="failed to get container status \"f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b\": rpc error: code = NotFound desc = could not find container \"f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b\": container with ID starting with f60ff76bd5d503a43eeab0f56fe595221c3d9509975e5cd8063cb989bff5565b not found: ID does not exist"
Sep 11 20:02:01.322573 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.322551    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"]
Sep 11 20:02:01.326070 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.326049    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f7fd24-9dc848bb6-hknpp"]
Sep 11 20:02:01.873118 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.873071    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d6c0804-cfe4-4690-9d9a-3bcbca306bdd" path="/var/lib/kubelet/pods/6d6c0804-cfe4-4690-9d9a-3bcbca306bdd/volumes"
Sep 11 20:02:01.873498 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:01.873481    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f705fabd-4ccc-4926-af16-43cdef7d4f27" path="/var/lib/kubelet/pods/f705fabd-4ccc-4926-af16-43cdef7d4f27/volumes"
Sep 11 20:02:03.644743 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:03.644709    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:02:12.569665 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.569628    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7500-88d377"]
Sep 11 20:02:12.570065 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.570044    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6c0804-cfe4-4690-9d9a-3bcbca306bdd" containerName="revoke-keys"
Sep 11 20:02:12.570065 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.570060    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6c0804-cfe4-4690-9d9a-3bcbca306bdd" containerName="revoke-keys"
Sep 11 20:02:12.570223 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.570075    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f705fabd-4ccc-4926-af16-43cdef7d4f27" containerName="payload-pre-processing"
Sep 11 20:02:12.570223 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.570082    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f705fabd-4ccc-4926-af16-43cdef7d4f27" containerName="payload-pre-processing"
Sep 11 20:02:12.570223 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.570149    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f705fabd-4ccc-4926-af16-43cdef7d4f27" containerName="payload-pre-processing"
Sep 11 20:02:12.570223 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.570157    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6c0804-cfe4-4690-9d9a-3bcbca306bdd" containerName="revoke-keys"
Sep 11 20:02:12.574374 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.574351    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7500-88d377"
Sep 11 20:02:12.576996 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.576971    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-76rlb\""
Sep 11 20:02:12.577664 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.577635    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7500-88d377"]
Sep 11 20:02:12.602185 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.602161    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgb95\" (UniqueName: \"kubernetes.io/projected/c6a56411-f982-4364-88c4-073590ca89aa-kube-api-access-rgb95\") pod \"test-curl-7500-88d377\" (UID: \"c6a56411-f982-4364-88c4-073590ca89aa\") " pod="opendatahub/test-curl-7500-88d377"
Sep 11 20:02:12.702763 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.702737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgb95\" (UniqueName: \"kubernetes.io/projected/c6a56411-f982-4364-88c4-073590ca89aa-kube-api-access-rgb95\") pod \"test-curl-7500-88d377\" (UID: \"c6a56411-f982-4364-88c4-073590ca89aa\") " pod="opendatahub/test-curl-7500-88d377"
Sep 11 20:02:12.711657 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.711636    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgb95\" (UniqueName: \"kubernetes.io/projected/c6a56411-f982-4364-88c4-073590ca89aa-kube-api-access-rgb95\") pod \"test-curl-7500-88d377\" (UID: \"c6a56411-f982-4364-88c4-073590ca89aa\") " pod="opendatahub/test-curl-7500-88d377"
Sep 11 20:02:12.885078 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:12.885051    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7500-88d377"
Sep 11 20:02:13.154667 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:13.154641    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7500-88d377"]
Sep 11 20:02:13.156432 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:02:13.156405    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6a56411_f982_4364_88c4_073590ca89aa.slice/crio-0424ac9b0f5a249c7f6b31c832903cf02d736ad579893e69fdd582c1106cd0ae WatchSource:0}: Error finding container 0424ac9b0f5a249c7f6b31c832903cf02d736ad579893e69fdd582c1106cd0ae: Status 404 returned error can't find the container with id 0424ac9b0f5a249c7f6b31c832903cf02d736ad579893e69fdd582c1106cd0ae
Sep 11 20:02:13.374295 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:13.374260    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7500-88d377" event={"ID":"c6a56411-f982-4364-88c4-073590ca89aa","Type":"ContainerStarted","Data":"29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875"}
Sep 11 20:02:13.374453 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:13.374305    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7500-88d377" event={"ID":"c6a56411-f982-4364-88c4-073590ca89aa","Type":"ContainerStarted","Data":"0424ac9b0f5a249c7f6b31c832903cf02d736ad579893e69fdd582c1106cd0ae"}
Sep 11 20:02:13.391544 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:13.391501    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7500-88d377" podStartSLOduration=1.3914864869999999 podStartE2EDuration="1.391486487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:02:13.389493429 +0000 UTC m=+1970.065430337" watchObservedRunningTime="2026-09-11 20:02:13.391486487 +0000 UTC m=+1970.067423395"
Sep 11 20:02:13.950321 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:13.950290    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7500-88d377"]
Sep 11 20:02:13.954871 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:13.954849    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7500-88d377"]
Sep 11 20:02:14.379876 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:02:14.379852    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7500-88d377" container="test-curl-7500-88d377"
Sep 11 20:02:15.268296 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:15.268267    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"]
Sep 11 20:02:15.385750 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:15.385710    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7500-88d377" podUID="c6a56411-f982-4364-88c4-073590ca89aa" containerName="test-curl-7500-88d377" containerID="cri-o://29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875" gracePeriod=2
Sep 11 20:02:15.388023 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:02:15.387999    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7500-88d377\" is forbidden: User \"system:node:ip-10-0-136-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-27.ec2.internal' and this object" podUID="c6a56411-f982-4364-88c4-073590ca89aa" pod="opendatahub/test-curl-7500-88d377"
Sep 11 20:02:16.169959 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:16.169867    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"]
Sep 11 20:02:16.843163 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:16.843131    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:02:17.558612 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:17.558589    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7500-88d377"
Sep 11 20:02:17.642025 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:17.641966    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgb95\" (UniqueName: \"kubernetes.io/projected/c6a56411-f982-4364-88c4-073590ca89aa-kube-api-access-rgb95\") pod \"c6a56411-f982-4364-88c4-073590ca89aa\" (UID: \"c6a56411-f982-4364-88c4-073590ca89aa\") "
Sep 11 20:02:17.643896 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:17.643862    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a56411-f982-4364-88c4-073590ca89aa-kube-api-access-rgb95" (OuterVolumeSpecName: "kube-api-access-rgb95") pod "c6a56411-f982-4364-88c4-073590ca89aa" (UID: "c6a56411-f982-4364-88c4-073590ca89aa"). InnerVolumeSpecName "kube-api-access-rgb95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:02:17.743149 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:17.743124    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgb95\" (UniqueName: \"kubernetes.io/projected/c6a56411-f982-4364-88c4-073590ca89aa-kube-api-access-rgb95\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:17.872897 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:17.872867    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a56411-f982-4364-88c4-073590ca89aa" path="/var/lib/kubelet/pods/c6a56411-f982-4364-88c4-073590ca89aa/volumes"
Sep 11 20:02:18.404217 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:18.404180    2571 generic.go:356] "Generic (PLEG): container finished" podID="c6a56411-f982-4364-88c4-073590ca89aa" containerID="29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875" exitCode=137
Sep 11 20:02:18.404364 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:18.404226    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7500-88d377"
Sep 11 20:02:18.404364 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:18.404274    2571 scope.go:117] "RemoveContainer" containerID="29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875"
Sep 11 20:02:18.412958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:18.412930    2571 scope.go:117] "RemoveContainer" containerID="29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875"
Sep 11 20:02:18.413252 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:02:18.413233    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875\": container with ID starting with 29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875 not found: ID does not exist" containerID="29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875"
Sep 11 20:02:18.413330 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:18.413264    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875"} err="failed to get container status \"29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875\": rpc error: code = NotFound desc = could not find container \"29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875\": container with ID starting with 29d1f1a6d826ddafb3f8ab99b028689e8bd860c4df3fcf397911f769fc092875 not found: ID does not exist"
Sep 11 20:02:23.944390 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.944356    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"]
Sep 11 20:02:23.944762 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.944658    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a56411-f982-4364-88c4-073590ca89aa" containerName="test-curl-7500-88d377"
Sep 11 20:02:23.944762 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.944670    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a56411-f982-4364-88c4-073590ca89aa" containerName="test-curl-7500-88d377"
Sep 11 20:02:23.944762 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.944723    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a56411-f982-4364-88c4-073590ca89aa" containerName="test-curl-7500-88d377"
Sep 11 20:02:23.948853 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.948826    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:23.951658 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.951637    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"sub-test-model-57d97d-kserve-self-signed-certs\""
Sep 11 20:02:23.958673 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.958650    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"]
Sep 11 20:02:23.990323 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.990296    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-tmp-dir\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:23.990436 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.990356    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-home\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:23.990436 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.990408    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-model-cache\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:23.990513 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.990437    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9990bbd4-b913-48dc-bab9-0de7088eff55-tls-certs\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:23.990513 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.990471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6j9p\" (UniqueName: \"kubernetes.io/projected/9990bbd4-b913-48dc-bab9-0de7088eff55-kube-api-access-m6j9p\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:23.990513 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:23.990489    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-dshm\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092081 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.091790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-home\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092081 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.091838    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-model-cache\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092081 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.091875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9990bbd4-b913-48dc-bab9-0de7088eff55-tls-certs\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092081 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.091922    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6j9p\" (UniqueName: \"kubernetes.io/projected/9990bbd4-b913-48dc-bab9-0de7088eff55-kube-api-access-m6j9p\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092081 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.091945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-dshm\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092081 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.091979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-tmp-dir\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092490 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.092380    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-tmp-dir\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092490 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.092442    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-model-cache\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.092605 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.092583    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-home\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.094489 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.094455    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-dshm\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.094778 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.094756    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9990bbd4-b913-48dc-bab9-0de7088eff55-tls-certs\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.101867 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.101842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6j9p\" (UniqueName: \"kubernetes.io/projected/9990bbd4-b913-48dc-bab9-0de7088eff55-kube-api-access-m6j9p\") pod \"sub-test-model-57d97d-kserve-679bc7bb55-zmm7b\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.258640 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.258576    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:24.530156 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:24.530130    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"]
Sep 11 20:02:24.532115 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:02:24.532077    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9990bbd4_b913_48dc_bab9_0de7088eff55.slice/crio-664aa5b0694a8af4324975ff9083f07163bc8670b502384346d1a25f13f3a819 WatchSource:0}: Error finding container 664aa5b0694a8af4324975ff9083f07163bc8670b502384346d1a25f13f3a819: Status 404 returned error can't find the container with id 664aa5b0694a8af4324975ff9083f07163bc8670b502384346d1a25f13f3a819
Sep 11 20:02:25.452348 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:25.452311    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b" event={"ID":"9990bbd4-b913-48dc-bab9-0de7088eff55","Type":"ContainerStarted","Data":"c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6"}
Sep 11 20:02:25.452724 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:25.452355    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b" event={"ID":"9990bbd4-b913-48dc-bab9-0de7088eff55","Type":"ContainerStarted","Data":"664aa5b0694a8af4324975ff9083f07163bc8670b502384346d1a25f13f3a819"}
Sep 11 20:02:25.470529 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:25.470475    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b" podStartSLOduration=2.470459301 podStartE2EDuration="2.470459301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:02:25.469490471 +0000 UTC m=+1982.145427379" watchObservedRunningTime="2026-09-11 20:02:25.470459301 +0000 UTC m=+1982.146396211"
Sep 11 20:02:27.246987 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:27.246950    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:02:30.286867 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.286811    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc" podUID="9dd62292-7ddd-4381-91de-32610918b7c7" containerName="main" containerID="cri-o://ab2a4cf5bb7fa465df05382ac82dff790f3f0d4eadfce78799f345bbeb1daf9b" gracePeriod=45
Sep 11 20:02:30.334695 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.334663    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc" podUID="9dd62292-7ddd-4381-91de-32610918b7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 11 20:02:30.480754 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.480724    2571 generic.go:356] "Generic (PLEG): container finished" podID="9dd62292-7ddd-4381-91de-32610918b7c7" containerID="ab2a4cf5bb7fa465df05382ac82dff790f3f0d4eadfce78799f345bbeb1daf9b" exitCode=0
Sep 11 20:02:30.480920 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.480793    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc" event={"ID":"9dd62292-7ddd-4381-91de-32610918b7c7","Type":"ContainerDied","Data":"ab2a4cf5bb7fa465df05382ac82dff790f3f0d4eadfce78799f345bbeb1daf9b"}
Sep 11 20:02:30.527467 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.527449    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 20:02:30.647000 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.646971    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n75t\" (UniqueName: \"kubernetes.io/projected/9dd62292-7ddd-4381-91de-32610918b7c7-kube-api-access-9n75t\") pod \"9dd62292-7ddd-4381-91de-32610918b7c7\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") "
Sep 11 20:02:30.647161 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.647073    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-dshm\") pod \"9dd62292-7ddd-4381-91de-32610918b7c7\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") "
Sep 11 20:02:30.647161 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.647134    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-home\") pod \"9dd62292-7ddd-4381-91de-32610918b7c7\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") "
Sep 11 20:02:30.647245 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.647207    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-model-cache\") pod \"9dd62292-7ddd-4381-91de-32610918b7c7\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") "
Sep 11 20:02:30.647291 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.647243    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd62292-7ddd-4381-91de-32610918b7c7-tls-certs\") pod \"9dd62292-7ddd-4381-91de-32610918b7c7\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") "
Sep 11 20:02:30.647291 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.647271    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-tmp-dir\") pod \"9dd62292-7ddd-4381-91de-32610918b7c7\" (UID: \"9dd62292-7ddd-4381-91de-32610918b7c7\") "
Sep 11 20:02:30.647463 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.647433    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "9dd62292-7ddd-4381-91de-32610918b7c7" (UID: "9dd62292-7ddd-4381-91de-32610918b7c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:30.647650 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.647446    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-home" (OuterVolumeSpecName: "home") pod "9dd62292-7ddd-4381-91de-32610918b7c7" (UID: "9dd62292-7ddd-4381-91de-32610918b7c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:30.647650 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.647574    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9dd62292-7ddd-4381-91de-32610918b7c7" (UID: "9dd62292-7ddd-4381-91de-32610918b7c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:30.649187 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.649162    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-dshm" (OuterVolumeSpecName: "dshm") pod "9dd62292-7ddd-4381-91de-32610918b7c7" (UID: "9dd62292-7ddd-4381-91de-32610918b7c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:30.649274 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.649221    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd62292-7ddd-4381-91de-32610918b7c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9dd62292-7ddd-4381-91de-32610918b7c7" (UID: "9dd62292-7ddd-4381-91de-32610918b7c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:02:30.649274 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.649230    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd62292-7ddd-4381-91de-32610918b7c7-kube-api-access-9n75t" (OuterVolumeSpecName: "kube-api-access-9n75t") pod "9dd62292-7ddd-4381-91de-32610918b7c7" (UID: "9dd62292-7ddd-4381-91de-32610918b7c7"). InnerVolumeSpecName "kube-api-access-9n75t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:02:30.748462 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.748439    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:30.748462 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.748462    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd62292-7ddd-4381-91de-32610918b7c7-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:30.748605 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.748471    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:30.748605 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.748479    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n75t\" (UniqueName: \"kubernetes.io/projected/9dd62292-7ddd-4381-91de-32610918b7c7-kube-api-access-9n75t\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:30.748605 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.748488    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:30.748605 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:30.748498    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd62292-7ddd-4381-91de-32610918b7c7-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:31.188315 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.188254    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g" podUID="77bc4a78-b956-45a2-b117-fe4efac876a7" containerName="main" containerID="cri-o://3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1" gracePeriod=45
Sep 11 20:02:31.428987 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.428965    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 20:02:31.487750 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.487672    2571 generic.go:356] "Generic (PLEG): container finished" podID="77bc4a78-b956-45a2-b117-fe4efac876a7" containerID="3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1" exitCode=0
Sep 11 20:02:31.487750 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.487705    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"
Sep 11 20:02:31.487972 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.487758    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g" event={"ID":"77bc4a78-b956-45a2-b117-fe4efac876a7","Type":"ContainerDied","Data":"3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1"}
Sep 11 20:02:31.487972 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.487802    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g" event={"ID":"77bc4a78-b956-45a2-b117-fe4efac876a7","Type":"ContainerDied","Data":"4b1e2088957528a1b88158dc535729ea6fd1a39983082f75a7641a2b8d887572"}
Sep 11 20:02:31.487972 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.487823    2571 scope.go:117] "RemoveContainer" containerID="3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1"
Sep 11 20:02:31.489317 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.489298    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc" event={"ID":"9dd62292-7ddd-4381-91de-32610918b7c7","Type":"ContainerDied","Data":"6bbbfcfcab7b22d9fb1891d0de28c215095eaf0eb3a83acdb39f172e391eea1c"}
Sep 11 20:02:31.489434 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.489334    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"
Sep 11 20:02:31.497625 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.497608    2571 scope.go:117] "RemoveContainer" containerID="3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1"
Sep 11 20:02:31.497929 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:02:31.497904    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1\": container with ID starting with 3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1 not found: ID does not exist" containerID="3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1"
Sep 11 20:02:31.498014 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.497933    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1"} err="failed to get container status \"3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1\": rpc error: code = NotFound desc = could not find container \"3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1\": container with ID starting with 3df4646a91462d598173570016fd927191cd121d19afd49d59e2d5dc77c674e1 not found: ID does not exist"
Sep 11 20:02:31.498014 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.497952    2571 scope.go:117] "RemoveContainer" containerID="ab2a4cf5bb7fa465df05382ac82dff790f3f0d4eadfce78799f345bbeb1daf9b"
Sep 11 20:02:31.511187 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.511166    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"]
Sep 11 20:02:31.515081 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.515061    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/auth-test-model-57d97d-kserve-559d9d9d99-9r7zc"]
Sep 11 20:02:31.554785 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.554763    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79qvj\" (UniqueName: \"kubernetes.io/projected/77bc4a78-b956-45a2-b117-fe4efac876a7-kube-api-access-79qvj\") pod \"77bc4a78-b956-45a2-b117-fe4efac876a7\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") "
Sep 11 20:02:31.554937 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.554798    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-home\") pod \"77bc4a78-b956-45a2-b117-fe4efac876a7\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") "
Sep 11 20:02:31.554937 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.554831    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-model-cache\") pod \"77bc4a78-b956-45a2-b117-fe4efac876a7\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") "
Sep 11 20:02:31.555024 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.554939    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-tmp-dir\") pod \"77bc4a78-b956-45a2-b117-fe4efac876a7\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") "
Sep 11 20:02:31.555024 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.554984    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-dshm\") pod \"77bc4a78-b956-45a2-b117-fe4efac876a7\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") "
Sep 11 20:02:31.555122 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.555023    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-home" (OuterVolumeSpecName: "home") pod "77bc4a78-b956-45a2-b117-fe4efac876a7" (UID: "77bc4a78-b956-45a2-b117-fe4efac876a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:31.555122 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.555034    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77bc4a78-b956-45a2-b117-fe4efac876a7-tls-certs\") pod \"77bc4a78-b956-45a2-b117-fe4efac876a7\" (UID: \"77bc4a78-b956-45a2-b117-fe4efac876a7\") "
Sep 11 20:02:31.555122 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.555042    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "77bc4a78-b956-45a2-b117-fe4efac876a7" (UID: "77bc4a78-b956-45a2-b117-fe4efac876a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:31.555236 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.555189    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77bc4a78-b956-45a2-b117-fe4efac876a7" (UID: "77bc4a78-b956-45a2-b117-fe4efac876a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:31.555342 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.555324    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:31.555397 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.555345    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:31.555397 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.555355    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:31.557036 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.557015    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77bc4a78-b956-45a2-b117-fe4efac876a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77bc4a78-b956-45a2-b117-fe4efac876a7" (UID: "77bc4a78-b956-45a2-b117-fe4efac876a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:02:31.557137 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.557018    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-dshm" (OuterVolumeSpecName: "dshm") pod "77bc4a78-b956-45a2-b117-fe4efac876a7" (UID: "77bc4a78-b956-45a2-b117-fe4efac876a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:31.557137 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.557031    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bc4a78-b956-45a2-b117-fe4efac876a7-kube-api-access-79qvj" (OuterVolumeSpecName: "kube-api-access-79qvj") pod "77bc4a78-b956-45a2-b117-fe4efac876a7" (UID: "77bc4a78-b956-45a2-b117-fe4efac876a7"). InnerVolumeSpecName "kube-api-access-79qvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:02:31.655968 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.655945    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77bc4a78-b956-45a2-b117-fe4efac876a7-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:31.655968 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.655966    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77bc4a78-b956-45a2-b117-fe4efac876a7-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:31.656142 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.655976    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qvj\" (UniqueName: \"kubernetes.io/projected/77bc4a78-b956-45a2-b117-fe4efac876a7-kube-api-access-79qvj\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:02:31.810300 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.810268    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"]
Sep 11 20:02:31.814073 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.814046    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/auth-test-model-8a4eb7-kserve-784599bff9-swh4g"]
Sep 11 20:02:31.872392 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.872366    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77bc4a78-b956-45a2-b117-fe4efac876a7" path="/var/lib/kubelet/pods/77bc4a78-b956-45a2-b117-fe4efac876a7/volumes"
Sep 11 20:02:31.872791 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:31.872775    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dd62292-7ddd-4381-91de-32610918b7c7" path="/var/lib/kubelet/pods/9dd62292-7ddd-4381-91de-32610918b7c7/volumes"
Sep 11 20:02:34.258844 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:34.258816    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:34.259383 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:34.259129    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:34.271325 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:34.271300    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:34.517875 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:34.517798    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:02:46.555715 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.555678    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"]
Sep 11 20:02:46.556197 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.555981    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd62292-7ddd-4381-91de-32610918b7c7" containerName="main"
Sep 11 20:02:46.556197 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.555993    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd62292-7ddd-4381-91de-32610918b7c7" containerName="main"
Sep 11 20:02:46.556197 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.556016    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bc4a78-b956-45a2-b117-fe4efac876a7" containerName="main"
Sep 11 20:02:46.556197 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.556021    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bc4a78-b956-45a2-b117-fe4efac876a7" containerName="main"
Sep 11 20:02:46.556197 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.556072    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bc4a78-b956-45a2-b117-fe4efac876a7" containerName="main"
Sep 11 20:02:46.556197 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.556081    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd62292-7ddd-4381-91de-32610918b7c7" containerName="main"
Sep 11 20:02:46.560414 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.560393    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.564584 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.564555    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"default-dockercfg-v47l5\""
Sep 11 20:02:46.564772 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.564748    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"sub-test-model-8a4eb7-kserve-self-signed-certs\""
Sep 11 20:02:46.564864 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.564842    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"kube-root-ca.crt\""
Sep 11 20:02:46.565294 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.564597    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"openshift-service-ca.crt\""
Sep 11 20:02:46.569635 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.569609    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"]
Sep 11 20:02:46.676915 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.676882    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-tmp-dir\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.677117 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.676924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-home\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.677117 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.676947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e191fa-132f-44ee-9cb5-677fb3ef9128-tls-certs\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.677117 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.676970    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-dshm\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.677117 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.677000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbjc\" (UniqueName: \"kubernetes.io/projected/67e191fa-132f-44ee-9cb5-677fb3ef9128-kube-api-access-fpbjc\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.677117 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.677026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-model-cache\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.777898 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.777868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-home\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.777898 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.777901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e191fa-132f-44ee-9cb5-677fb3ef9128-tls-certs\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.778171 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.777925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-dshm\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.778171 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.777956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbjc\" (UniqueName: \"kubernetes.io/projected/67e191fa-132f-44ee-9cb5-677fb3ef9128-kube-api-access-fpbjc\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.778171 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.777990    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-model-cache\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.778171 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.778020    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-tmp-dir\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.778460 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.778355    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-home\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.778460 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.778392    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-model-cache\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.778529 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.778471    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-tmp-dir\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.780199 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.780181    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-dshm\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.780377 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.780358    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e191fa-132f-44ee-9cb5-677fb3ef9128-tls-certs\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.787449 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.787424    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbjc\" (UniqueName: \"kubernetes.io/projected/67e191fa-132f-44ee-9cb5-677fb3ef9128-kube-api-access-fpbjc\") pod \"sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:46.872297 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:46.872261    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:47.151067 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:47.151040    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"]
Sep 11 20:02:47.153242 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:02:47.153213    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e191fa_132f_44ee_9cb5_677fb3ef9128.slice/crio-506f72555f70637d907d1c4c8dd0d567b8237c9576ba363f377372f6d87ae4fc WatchSource:0}: Error finding container 506f72555f70637d907d1c4c8dd0d567b8237c9576ba363f377372f6d87ae4fc: Status 404 returned error can't find the container with id 506f72555f70637d907d1c4c8dd0d567b8237c9576ba363f377372f6d87ae4fc
Sep 11 20:02:47.576546 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:47.576451    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv" event={"ID":"67e191fa-132f-44ee-9cb5-677fb3ef9128","Type":"ContainerStarted","Data":"92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391"}
Sep 11 20:02:47.576546 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:47.576496    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv" event={"ID":"67e191fa-132f-44ee-9cb5-677fb3ef9128","Type":"ContainerStarted","Data":"506f72555f70637d907d1c4c8dd0d567b8237c9576ba363f377372f6d87ae4fc"}
Sep 11 20:02:47.600608 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:47.600551    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv" podStartSLOduration=1.600532908 podStartE2EDuration="1.600532908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:02:47.597480138 +0000 UTC m=+2004.273417047" watchObservedRunningTime="2026-09-11 20:02:47.600532908 +0000 UTC m=+2004.276469816"
Sep 11 20:02:50.350076 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:50.349984    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:02:56.872736 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:56.872697    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:56.873266 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:56.872758    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:56.885279 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:56.885253    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:02:57.639593 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:02:57.639563    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:03:08.744689 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:08.744642    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:03:21.841213 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:21.841176    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:03:25.345659 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.345626    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7500-739fde"]
Sep 11 20:03:25.350074 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.350056    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7500-739fde"
Sep 11 20:03:25.352739 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.352719    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-76rlb\""
Sep 11 20:03:25.357481 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.357449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxl8\" (UniqueName: \"kubernetes.io/projected/35dddd59-fb1f-48cd-88a3-044eddfad8b8-kube-api-access-kcxl8\") pod \"test-curl-7500-739fde\" (UID: \"35dddd59-fb1f-48cd-88a3-044eddfad8b8\") " pod="opendatahub/test-curl-7500-739fde"
Sep 11 20:03:25.358674 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.358652    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7500-739fde"]
Sep 11 20:03:25.458155 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.458123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxl8\" (UniqueName: \"kubernetes.io/projected/35dddd59-fb1f-48cd-88a3-044eddfad8b8-kube-api-access-kcxl8\") pod \"test-curl-7500-739fde\" (UID: \"35dddd59-fb1f-48cd-88a3-044eddfad8b8\") " pod="opendatahub/test-curl-7500-739fde"
Sep 11 20:03:25.466063 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.466035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxl8\" (UniqueName: \"kubernetes.io/projected/35dddd59-fb1f-48cd-88a3-044eddfad8b8-kube-api-access-kcxl8\") pod \"test-curl-7500-739fde\" (UID: \"35dddd59-fb1f-48cd-88a3-044eddfad8b8\") " pod="opendatahub/test-curl-7500-739fde"
Sep 11 20:03:25.660145 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.660096    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7500-739fde"
Sep 11 20:03:25.932706 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:25.930331    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7500-739fde"]
Sep 11 20:03:25.934230 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:03:25.934200    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35dddd59_fb1f_48cd_88a3_044eddfad8b8.slice/crio-e43b47431d9019a12ae0f0bbe3392f83061f09cdabf65cddc00808bb637765ec WatchSource:0}: Error finding container e43b47431d9019a12ae0f0bbe3392f83061f09cdabf65cddc00808bb637765ec: Status 404 returned error can't find the container with id e43b47431d9019a12ae0f0bbe3392f83061f09cdabf65cddc00808bb637765ec
Sep 11 20:03:26.778557 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:26.778516    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7500-739fde" event={"ID":"35dddd59-fb1f-48cd-88a3-044eddfad8b8","Type":"ContainerStarted","Data":"38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87"}
Sep 11 20:03:26.778925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:26.778564    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7500-739fde" event={"ID":"35dddd59-fb1f-48cd-88a3-044eddfad8b8","Type":"ContainerStarted","Data":"e43b47431d9019a12ae0f0bbe3392f83061f09cdabf65cddc00808bb637765ec"}
Sep 11 20:03:26.794284 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:26.794239    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7500-739fde" podStartSLOduration=1.794224689 podStartE2EDuration="1.794224689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:26.792252771 +0000 UTC m=+2043.468189679" watchObservedRunningTime="2026-09-11 20:03:26.794224689 +0000 UTC m=+2043.470161594"
Sep 11 20:03:27.349864 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:27.349825    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7500-739fde"]
Sep 11 20:03:27.352463 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:27.352437    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7500-739fde"]
Sep 11 20:03:27.783733 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:03:27.783701    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7500-739fde" container="test-curl-7500-739fde"
Sep 11 20:03:28.788566 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:28.788528    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7500-739fde" podUID="35dddd59-fb1f-48cd-88a3-044eddfad8b8" containerName="test-curl-7500-739fde" containerID="cri-o://38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87" gracePeriod=2
Sep 11 20:03:30.959403 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:30.959381    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7500-739fde"
Sep 11 20:03:31.002295 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.002249    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcxl8\" (UniqueName: \"kubernetes.io/projected/35dddd59-fb1f-48cd-88a3-044eddfad8b8-kube-api-access-kcxl8\") pod \"35dddd59-fb1f-48cd-88a3-044eddfad8b8\" (UID: \"35dddd59-fb1f-48cd-88a3-044eddfad8b8\") "
Sep 11 20:03:31.004238 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.004212    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35dddd59-fb1f-48cd-88a3-044eddfad8b8-kube-api-access-kcxl8" (OuterVolumeSpecName: "kube-api-access-kcxl8") pod "35dddd59-fb1f-48cd-88a3-044eddfad8b8" (UID: "35dddd59-fb1f-48cd-88a3-044eddfad8b8"). InnerVolumeSpecName "kube-api-access-kcxl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:03:31.103502 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.103408    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxl8\" (UniqueName: \"kubernetes.io/projected/35dddd59-fb1f-48cd-88a3-044eddfad8b8-kube-api-access-kcxl8\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:31.467045 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.467008    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"]
Sep 11 20:03:31.806503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.806410    2571 generic.go:356] "Generic (PLEG): container finished" podID="35dddd59-fb1f-48cd-88a3-044eddfad8b8" containerID="38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87" exitCode=137
Sep 11 20:03:31.806503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.806465    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7500-739fde"
Sep 11 20:03:31.806688 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.806469    2571 scope.go:117] "RemoveContainer" containerID="38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87"
Sep 11 20:03:31.815906 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.815889    2571 scope.go:117] "RemoveContainer" containerID="38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87"
Sep 11 20:03:31.816155 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:03:31.816135    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87\": container with ID starting with 38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87 not found: ID does not exist" containerID="38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87"
Sep 11 20:03:31.816209 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.816164    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87"} err="failed to get container status \"38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87\": rpc error: code = NotFound desc = could not find container \"38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87\": container with ID starting with 38897f2d3acc0ee5c21ab4366a94a6734d330fb5de5682cdc6eceeed1c249c87 not found: ID does not exist"
Sep 11 20:03:31.817417 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:03:31.817390    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7500-739fde\" is forbidden: User \"system:node:ip-10-0-136-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-27.ec2.internal' and this object" podUID="35dddd59-fb1f-48cd-88a3-044eddfad8b8" pod="opendatahub/test-curl-7500-739fde"
Sep 11 20:03:31.872722 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:31.872691    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35dddd59-fb1f-48cd-88a3-044eddfad8b8" path="/var/lib/kubelet/pods/35dddd59-fb1f-48cd-88a3-044eddfad8b8/volumes"
Sep 11 20:03:32.461141 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:32.461095    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"]
Sep 11 20:03:34.755747 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:34.755710    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:03:46.486122 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.486003    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b" podUID="9990bbd4-b913-48dc-bab9-0de7088eff55" containerName="main" containerID="cri-o://c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6" gracePeriod=45
Sep 11 20:03:46.729776 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.729755    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:03:46.830658 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.830550    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-model-cache\") pod \"9990bbd4-b913-48dc-bab9-0de7088eff55\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") "
Sep 11 20:03:46.830658 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.830618    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-home\") pod \"9990bbd4-b913-48dc-bab9-0de7088eff55\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") "
Sep 11 20:03:46.830974 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.830672    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6j9p\" (UniqueName: \"kubernetes.io/projected/9990bbd4-b913-48dc-bab9-0de7088eff55-kube-api-access-m6j9p\") pod \"9990bbd4-b913-48dc-bab9-0de7088eff55\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") "
Sep 11 20:03:46.830974 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.830708    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-dshm\") pod \"9990bbd4-b913-48dc-bab9-0de7088eff55\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") "
Sep 11 20:03:46.830974 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.830733    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9990bbd4-b913-48dc-bab9-0de7088eff55-tls-certs\") pod \"9990bbd4-b913-48dc-bab9-0de7088eff55\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") "
Sep 11 20:03:46.830974 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.830764    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-tmp-dir\") pod \"9990bbd4-b913-48dc-bab9-0de7088eff55\" (UID: \"9990bbd4-b913-48dc-bab9-0de7088eff55\") "
Sep 11 20:03:46.830974 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.830887    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-home" (OuterVolumeSpecName: "home") pod "9990bbd4-b913-48dc-bab9-0de7088eff55" (UID: "9990bbd4-b913-48dc-bab9-0de7088eff55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:03:46.830974 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.830900    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-model-cache" (OuterVolumeSpecName: "model-cache") pod "9990bbd4-b913-48dc-bab9-0de7088eff55" (UID: "9990bbd4-b913-48dc-bab9-0de7088eff55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:03:46.831284 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.831029    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:46.831284 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.831051    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:46.831284 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.831078    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9990bbd4-b913-48dc-bab9-0de7088eff55" (UID: "9990bbd4-b913-48dc-bab9-0de7088eff55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:03:46.832898 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.832873    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-dshm" (OuterVolumeSpecName: "dshm") pod "9990bbd4-b913-48dc-bab9-0de7088eff55" (UID: "9990bbd4-b913-48dc-bab9-0de7088eff55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:03:46.832995 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.832929    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9990bbd4-b913-48dc-bab9-0de7088eff55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9990bbd4-b913-48dc-bab9-0de7088eff55" (UID: "9990bbd4-b913-48dc-bab9-0de7088eff55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:03:46.832995 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.832942    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9990bbd4-b913-48dc-bab9-0de7088eff55-kube-api-access-m6j9p" (OuterVolumeSpecName: "kube-api-access-m6j9p") pod "9990bbd4-b913-48dc-bab9-0de7088eff55" (UID: "9990bbd4-b913-48dc-bab9-0de7088eff55"). InnerVolumeSpecName "kube-api-access-m6j9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:03:46.886334 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.886300    2571 generic.go:356] "Generic (PLEG): container finished" podID="9990bbd4-b913-48dc-bab9-0de7088eff55" containerID="c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6" exitCode=0
Sep 11 20:03:46.886549 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.886352    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"
Sep 11 20:03:46.886549 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.886390    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b" event={"ID":"9990bbd4-b913-48dc-bab9-0de7088eff55","Type":"ContainerDied","Data":"c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6"}
Sep 11 20:03:46.886549 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.886446    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b" event={"ID":"9990bbd4-b913-48dc-bab9-0de7088eff55","Type":"ContainerDied","Data":"664aa5b0694a8af4324975ff9083f07163bc8670b502384346d1a25f13f3a819"}
Sep 11 20:03:46.886549 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.886465    2571 scope.go:117] "RemoveContainer" containerID="c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6"
Sep 11 20:03:46.895568 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.895549    2571 scope.go:117] "RemoveContainer" containerID="c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6"
Sep 11 20:03:46.895818 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:03:46.895801    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6\": container with ID starting with c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6 not found: ID does not exist" containerID="c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6"
Sep 11 20:03:46.895866 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.895826    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6"} err="failed to get container status \"c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6\": rpc error: code = NotFound desc = could not find container \"c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6\": container with ID starting with c2a884ad7ca87ff911be30d5b9e45f4e55ece9e724b6777a1e8d09f0d56a8fd6 not found: ID does not exist"
Sep 11 20:03:46.908459 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.908431    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"]
Sep 11 20:03:46.914486 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.914462    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/sub-test-model-57d97d-kserve-679bc7bb55-zmm7b"]
Sep 11 20:03:46.931933 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.931913    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6j9p\" (UniqueName: \"kubernetes.io/projected/9990bbd4-b913-48dc-bab9-0de7088eff55-kube-api-access-m6j9p\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:46.931933 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.931931    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:46.932051 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.931940    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9990bbd4-b913-48dc-bab9-0de7088eff55-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:46.932051 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:46.931948    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9990bbd4-b913-48dc-bab9-0de7088eff55-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:47.479755 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.479682    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv" podUID="67e191fa-132f-44ee-9cb5-677fb3ef9128" containerName="main" containerID="cri-o://92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391" gracePeriod=45
Sep 11 20:03:47.629456 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.629418    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv" podUID="67e191fa-132f-44ee-9cb5-677fb3ef9128" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 11 20:03:47.712363 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.712336    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:03:47.839562 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.839462    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e191fa-132f-44ee-9cb5-677fb3ef9128-tls-certs\") pod \"67e191fa-132f-44ee-9cb5-677fb3ef9128\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") "
Sep 11 20:03:47.839562 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.839548    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpbjc\" (UniqueName: \"kubernetes.io/projected/67e191fa-132f-44ee-9cb5-677fb3ef9128-kube-api-access-fpbjc\") pod \"67e191fa-132f-44ee-9cb5-677fb3ef9128\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") "
Sep 11 20:03:47.839819 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.839576    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-tmp-dir\") pod \"67e191fa-132f-44ee-9cb5-677fb3ef9128\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") "
Sep 11 20:03:47.839819 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.839602    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-model-cache\") pod \"67e191fa-132f-44ee-9cb5-677fb3ef9128\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") "
Sep 11 20:03:47.839819 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.839634    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-dshm\") pod \"67e191fa-132f-44ee-9cb5-677fb3ef9128\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") "
Sep 11 20:03:47.839819 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.839667    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-home\") pod \"67e191fa-132f-44ee-9cb5-677fb3ef9128\" (UID: \"67e191fa-132f-44ee-9cb5-677fb3ef9128\") "
Sep 11 20:03:47.840028 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.839884    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-model-cache" (OuterVolumeSpecName: "model-cache") pod "67e191fa-132f-44ee-9cb5-677fb3ef9128" (UID: "67e191fa-132f-44ee-9cb5-677fb3ef9128"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:03:47.840028 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.839891    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67e191fa-132f-44ee-9cb5-677fb3ef9128" (UID: "67e191fa-132f-44ee-9cb5-677fb3ef9128"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:03:47.840156 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.840062    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-home" (OuterVolumeSpecName: "home") pod "67e191fa-132f-44ee-9cb5-677fb3ef9128" (UID: "67e191fa-132f-44ee-9cb5-677fb3ef9128"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:03:47.841679 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.841650    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e191fa-132f-44ee-9cb5-677fb3ef9128-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67e191fa-132f-44ee-9cb5-677fb3ef9128" (UID: "67e191fa-132f-44ee-9cb5-677fb3ef9128"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:03:47.841815 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.841795    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-dshm" (OuterVolumeSpecName: "dshm") pod "67e191fa-132f-44ee-9cb5-677fb3ef9128" (UID: "67e191fa-132f-44ee-9cb5-677fb3ef9128"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:03:47.841863 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.841832    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e191fa-132f-44ee-9cb5-677fb3ef9128-kube-api-access-fpbjc" (OuterVolumeSpecName: "kube-api-access-fpbjc") pod "67e191fa-132f-44ee-9cb5-677fb3ef9128" (UID: "67e191fa-132f-44ee-9cb5-677fb3ef9128"). InnerVolumeSpecName "kube-api-access-fpbjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:03:47.872893 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.872860    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9990bbd4-b913-48dc-bab9-0de7088eff55" path="/var/lib/kubelet/pods/9990bbd4-b913-48dc-bab9-0de7088eff55/volumes"
Sep 11 20:03:47.893676 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.893646    2571 generic.go:356] "Generic (PLEG): container finished" podID="67e191fa-132f-44ee-9cb5-677fb3ef9128" containerID="92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391" exitCode=0
Sep 11 20:03:47.893848 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.893687    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"
Sep 11 20:03:47.893848 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.893734    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv" event={"ID":"67e191fa-132f-44ee-9cb5-677fb3ef9128","Type":"ContainerDied","Data":"92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391"}
Sep 11 20:03:47.893848 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.893775    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv" event={"ID":"67e191fa-132f-44ee-9cb5-677fb3ef9128","Type":"ContainerDied","Data":"506f72555f70637d907d1c4c8dd0d567b8237c9576ba363f377372f6d87ae4fc"}
Sep 11 20:03:47.893848 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.893792    2571 scope.go:117] "RemoveContainer" containerID="92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391"
Sep 11 20:03:47.902629 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.902608    2571 scope.go:117] "RemoveContainer" containerID="92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391"
Sep 11 20:03:47.902914 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:03:47.902892    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391\": container with ID starting with 92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391 not found: ID does not exist" containerID="92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391"
Sep 11 20:03:47.902986 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.902919    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391"} err="failed to get container status \"92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391\": rpc error: code = NotFound desc = could not find container \"92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391\": container with ID starting with 92ec868394dd9e468a633b0f2855436d0f6f53bb7fae87a1f5cb8f5982b55391 not found: ID does not exist"
Sep 11 20:03:47.910800 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.910773    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"]
Sep 11 20:03:47.916726 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.916702    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/sub-test-model-8a4eb7-kserve-79d6ff4697-fcmzv"]
Sep 11 20:03:47.940873 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.940837    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:47.941078 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.940947    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e191fa-132f-44ee-9cb5-677fb3ef9128-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:47.941078 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.940967    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpbjc\" (UniqueName: \"kubernetes.io/projected/67e191fa-132f-44ee-9cb5-677fb3ef9128-kube-api-access-fpbjc\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:47.941078 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.940977    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:47.941078 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.940986    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:47.941078 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:47.940994    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e191fa-132f-44ee-9cb5-677fb3ef9128-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:03:49.873283 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:49.873249    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e191fa-132f-44ee-9cb5-677fb3ef9128" path="/var/lib/kubelet/pods/67e191fa-132f-44ee-9cb5-677fb3ef9128/volumes"
Sep 11 20:03:50.555811 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.555692    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"]
Sep 11 20:03:50.556554 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556508    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e191fa-132f-44ee-9cb5-677fb3ef9128" containerName="main"
Sep 11 20:03:50.556727 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556537    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e191fa-132f-44ee-9cb5-677fb3ef9128" containerName="main"
Sep 11 20:03:50.556798 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556732    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9990bbd4-b913-48dc-bab9-0de7088eff55" containerName="main"
Sep 11 20:03:50.556798 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556741    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9990bbd4-b913-48dc-bab9-0de7088eff55" containerName="main"
Sep 11 20:03:50.556798 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556751    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35dddd59-fb1f-48cd-88a3-044eddfad8b8" containerName="test-curl-7500-739fde"
Sep 11 20:03:50.556798 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556758    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="35dddd59-fb1f-48cd-88a3-044eddfad8b8" containerName="test-curl-7500-739fde"
Sep 11 20:03:50.556997 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556822    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e191fa-132f-44ee-9cb5-677fb3ef9128" containerName="main"
Sep 11 20:03:50.556997 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556829    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9990bbd4-b913-48dc-bab9-0de7088eff55" containerName="main"
Sep 11 20:03:50.556997 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.556836    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="35dddd59-fb1f-48cd-88a3-044eddfad8b8" containerName="test-curl-7500-739fde"
Sep 11 20:03:50.562745 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.562692    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"]
Sep 11 20:03:50.562951 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.562804    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.566090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.566052    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"kube-root-ca.crt\""
Sep 11 20:03:50.566794 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.566771    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"default-dockercfg-s2gvw\""
Sep 11 20:03:50.568040 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.567228    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"openshift-service-ca.crt\""
Sep 11 20:03:50.568169 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.567288    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-57d97d\"/\"rate-test-model-57d97d-kserve-self-signed-certs\""
Sep 11 20:03:50.663862 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.663819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-dshm\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.664054 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.663872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w66p\" (UniqueName: \"kubernetes.io/projected/d8541d27-56d1-4b97-9cec-0557d180a424-kube-api-access-9w66p\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.664054 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.663932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-model-cache\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.664054 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.663975    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-home\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.664054 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.663994    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-tmp-dir\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.664054 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.664011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8541d27-56d1-4b97-9cec-0557d180a424-tls-certs\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765221 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765163    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-model-cache\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765221 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765231    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-home\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765509 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-tmp-dir\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765509 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765275    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8541d27-56d1-4b97-9cec-0557d180a424-tls-certs\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765509 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-dshm\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765509 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w66p\" (UniqueName: \"kubernetes.io/projected/d8541d27-56d1-4b97-9cec-0557d180a424-kube-api-access-9w66p\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765722 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-model-cache\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765722 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765688    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-tmp-dir\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.765801 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.765718    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-home\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.767786 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.767762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-dshm\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.767966 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.767948    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8541d27-56d1-4b97-9cec-0557d180a424-tls-certs\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.774235 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.774194    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w66p\" (UniqueName: \"kubernetes.io/projected/d8541d27-56d1-4b97-9cec-0557d180a424-kube-api-access-9w66p\") pod \"rate-test-model-57d97d-kserve-856df488d7-9n89m\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") " pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:50.876150 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:50.876079    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:03:51.149915 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:51.149890    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"]
Sep 11 20:03:51.151919 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:03:51.151889    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8541d27_56d1_4b97_9cec_0557d180a424.slice/crio-76f872e85f4875e30f98a47878ee40f011a026ebbd2afbb4e49a2283fd47a0c4 WatchSource:0}: Error finding container 76f872e85f4875e30f98a47878ee40f011a026ebbd2afbb4e49a2283fd47a0c4: Status 404 returned error can't find the container with id 76f872e85f4875e30f98a47878ee40f011a026ebbd2afbb4e49a2283fd47a0c4
Sep 11 20:03:51.919003 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:51.918962    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m" event={"ID":"d8541d27-56d1-4b97-9cec-0557d180a424","Type":"ContainerStarted","Data":"a9a98f9e324d6dbf494c03c4f47734971145fee4a43d2a0b2ab58a92f8d8a3a0"}
Sep 11 20:03:51.919003 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:51.919007    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m" event={"ID":"d8541d27-56d1-4b97-9cec-0557d180a424","Type":"ContainerStarted","Data":"76f872e85f4875e30f98a47878ee40f011a026ebbd2afbb4e49a2283fd47a0c4"}
Sep 11 20:03:51.940467 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:51.940417    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m" podStartSLOduration=1.940398461 podStartE2EDuration="1.940398461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:51.938748474 +0000 UTC m=+2068.614685383" watchObservedRunningTime="2026-09-11 20:03:51.940398461 +0000 UTC m=+2068.616335370"
Sep 11 20:03:53.748602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:03:53.748562    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:04:00.877277 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:00.877237    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:04:00.877277 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:00.877287    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:04:00.889822 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:00.889797    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:04:00.976699 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:00.976668    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:04:08.634938 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.634897    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"]
Sep 11 20:04:08.638094 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.638074    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.640940 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.640914    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"openshift-service-ca.crt\""
Sep 11 20:04:08.641087 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.640916    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"kube-root-ca.crt\""
Sep 11 20:04:08.642085 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.642065    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"rate-test-model-8a4eb7-kserve-self-signed-certs\""
Sep 11 20:04:08.642085 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.642081    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8a4eb7\"/\"default-dockercfg-v47l5\""
Sep 11 20:04:08.649882 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.649857    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"]
Sep 11 20:04:08.722467 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.722433    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-dshm\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.722655 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.722475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74854437-4fd9-48f7-8afb-801623a34297-tls-certs\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.722655 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.722506    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-tmp-dir\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.722655 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.722597    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-model-cache\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.722655 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.722647    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kznkl\" (UniqueName: \"kubernetes.io/projected/74854437-4fd9-48f7-8afb-801623a34297-kube-api-access-kznkl\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.722827 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.722712    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-home\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823332 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823293    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-tmp-dir\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823524 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-model-cache\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823524 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kznkl\" (UniqueName: \"kubernetes.io/projected/74854437-4fd9-48f7-8afb-801623a34297-kube-api-access-kznkl\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823524 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-home\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823524 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-dshm\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823524 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74854437-4fd9-48f7-8afb-801623a34297-tls-certs\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823806 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823764    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-tmp-dir\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823863 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823818    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-home\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.823863 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.823814    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-model-cache\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.825765 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.825739    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-dshm\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.826041 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.826020    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74854437-4fd9-48f7-8afb-801623a34297-tls-certs\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.831274 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.831255    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kznkl\" (UniqueName: \"kubernetes.io/projected/74854437-4fd9-48f7-8afb-801623a34297-kube-api-access-kznkl\") pod \"rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") " pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:08.948245 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:08.948153    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:09.221397 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:09.221366    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"]
Sep 11 20:04:09.222831 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:04:09.222799    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74854437_4fd9_48f7_8afb_801623a34297.slice/crio-87ad5ec2d339c8332abaae93c817057d96761104e54648475bcd78bc3ed219f2 WatchSource:0}: Error finding container 87ad5ec2d339c8332abaae93c817057d96761104e54648475bcd78bc3ed219f2: Status 404 returned error can't find the container with id 87ad5ec2d339c8332abaae93c817057d96761104e54648475bcd78bc3ed219f2
Sep 11 20:04:10.014287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:10.014241    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww" event={"ID":"74854437-4fd9-48f7-8afb-801623a34297","Type":"ContainerStarted","Data":"d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605"}
Sep 11 20:04:10.014287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:10.014294    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww" event={"ID":"74854437-4fd9-48f7-8afb-801623a34297","Type":"ContainerStarted","Data":"87ad5ec2d339c8332abaae93c817057d96761104e54648475bcd78bc3ed219f2"}
Sep 11 20:04:10.034186 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:10.034138    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww" podStartSLOduration=2.034120979 podStartE2EDuration="2.034120979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:04:10.031493881 +0000 UTC m=+2086.707430789" watchObservedRunningTime="2026-09-11 20:04:10.034120979 +0000 UTC m=+2086.710057889"
Sep 11 20:04:12.049397 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:12.049361    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:04:18.948448 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:18.948405    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:18.949000 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:18.948533    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:18.961260 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:18.961231    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:19.074705 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:19.074667    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:04:24.140020 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:24.139985    2571 scope.go:117] "RemoveContainer" containerID="7bdadaaf55a74cd3b12e63c918bede43d727e9ef73e86e6bd07728d9ba7c180f"
Sep 11 20:04:25.246773 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:25.246728    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:04:33.144833 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:33.144794    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:04:42.442067 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:42.442027    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:04:46.102014 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:46.101979    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"]
Sep 11 20:04:47.240611 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:47.240573    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"]
Sep 11 20:04:47.952627 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:47.952594    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"]
Sep 11 20:04:47.956176 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:47.956155    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:47.958910 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:47.958887    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-96e341-openshift-default-dockercfg-xlmxm\""
Sep 11 20:04:47.959041 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:47.958892    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 20:04:47.966532 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:47.966507    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"]
Sep 11 20:04:48.036183 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036144    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-socket\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.036183 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036184    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmk8k\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-kube-api-access-wmk8k\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.036422 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036209    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-credential-socket\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.036422 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-certs\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.036422 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036300    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884f0e-1142-4530-a781-a4dfebb30804-istiod-ca-cert\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.036422 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-data\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.036422 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036384    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-envoy\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.036422 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036399    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-istio-token\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.036422 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.036420    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884f0e-1142-4530-a781-a4dfebb30804-istio-podinfo\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137399 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137358    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-data\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137399 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137395    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-envoy\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137638 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-istio-token\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137638 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884f0e-1142-4530-a781-a4dfebb30804-istio-podinfo\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137638 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137592    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-socket\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137811 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmk8k\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-kube-api-access-wmk8k\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137811 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137796    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-credential-socket\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137825    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-data\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-certs\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.137925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137871    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-socket\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.138129 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.137923    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884f0e-1142-4530-a781-a4dfebb30804-istiod-ca-cert\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.138218 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.138194    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-credential-socket\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.138293 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.138272    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-certs\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.138528 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.138510    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884f0e-1142-4530-a781-a4dfebb30804-istiod-ca-cert\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.139760 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.139737    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-envoy\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.140004 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.139989    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884f0e-1142-4530-a781-a4dfebb30804-istio-podinfo\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.145572 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.145547    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmk8k\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-kube-api-access-wmk8k\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.145660 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.145567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-istio-token\") pod \"e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") " pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.266860 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.266774    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:48.546565 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.546538    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"]
Sep 11 20:04:48.548813 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:04:48.548785    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03884f0e_1142_4530_a781_a4dfebb30804.slice/crio-05a43d308293e85d285dbb71465928e0c190827dc5dcc43ee35e6b1dd9a2d88b WatchSource:0}: Error finding container 05a43d308293e85d285dbb71465928e0c190827dc5dcc43ee35e6b1dd9a2d88b: Status 404 returned error can't find the container with id 05a43d308293e85d285dbb71465928e0c190827dc5dcc43ee35e6b1dd9a2d88b
Sep 11 20:04:48.551464 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.551427    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:04:48.551555 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.551499    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:04:48.551601 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:48.551552    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:04:49.231661 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:49.231627    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" event={"ID":"03884f0e-1142-4530-a781-a4dfebb30804","Type":"ContainerStarted","Data":"6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c"}
Sep 11 20:04:49.231919 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:49.231688    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" event={"ID":"03884f0e-1142-4530-a781-a4dfebb30804","Type":"ContainerStarted","Data":"05a43d308293e85d285dbb71465928e0c190827dc5dcc43ee35e6b1dd9a2d88b"}
Sep 11 20:04:49.254943 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:49.254887    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" podStartSLOduration=2.254870027 podStartE2EDuration="2.254870027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:04:49.253982177 +0000 UTC m=+2125.929919086" watchObservedRunningTime="2026-09-11 20:04:49.254870027 +0000 UTC m=+2125.930806935"
Sep 11 20:04:49.267285 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:49.267250    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:49.272058 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:49.272032    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:50.237442 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:50.237416    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:50.238286 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:50.238265    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:04:53.451229 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:04:53.451195    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:05:01.130501 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.130418    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m" podUID="d8541d27-56d1-4b97-9cec-0557d180a424" containerName="main" containerID="cri-o://a9a98f9e324d6dbf494c03c4f47734971145fee4a43d2a0b2ab58a92f8d8a3a0" gracePeriod=45
Sep 11 20:05:01.299712 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.299678    2571 generic.go:356] "Generic (PLEG): container finished" podID="d8541d27-56d1-4b97-9cec-0557d180a424" containerID="a9a98f9e324d6dbf494c03c4f47734971145fee4a43d2a0b2ab58a92f8d8a3a0" exitCode=0
Sep 11 20:05:01.299850 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.299744    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m" event={"ID":"d8541d27-56d1-4b97-9cec-0557d180a424","Type":"ContainerDied","Data":"a9a98f9e324d6dbf494c03c4f47734971145fee4a43d2a0b2ab58a92f8d8a3a0"}
Sep 11 20:05:01.382359 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.382284    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:05:01.453562 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.453518    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-dshm\") pod \"d8541d27-56d1-4b97-9cec-0557d180a424\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") "
Sep 11 20:05:01.453764 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.453572    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-model-cache\") pod \"d8541d27-56d1-4b97-9cec-0557d180a424\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") "
Sep 11 20:05:01.453764 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.453648    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8541d27-56d1-4b97-9cec-0557d180a424-tls-certs\") pod \"d8541d27-56d1-4b97-9cec-0557d180a424\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") "
Sep 11 20:05:01.453764 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.453696    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w66p\" (UniqueName: \"kubernetes.io/projected/d8541d27-56d1-4b97-9cec-0557d180a424-kube-api-access-9w66p\") pod \"d8541d27-56d1-4b97-9cec-0557d180a424\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") "
Sep 11 20:05:01.453764 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.453730    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-home\") pod \"d8541d27-56d1-4b97-9cec-0557d180a424\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") "
Sep 11 20:05:01.453993 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.453849    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8541d27-56d1-4b97-9cec-0557d180a424" (UID: "d8541d27-56d1-4b97-9cec-0557d180a424"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:01.453993 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.453901    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-tmp-dir\") pod \"d8541d27-56d1-4b97-9cec-0557d180a424\" (UID: \"d8541d27-56d1-4b97-9cec-0557d180a424\") "
Sep 11 20:05:01.453993 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.453958    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-home" (OuterVolumeSpecName: "home") pod "d8541d27-56d1-4b97-9cec-0557d180a424" (UID: "d8541d27-56d1-4b97-9cec-0557d180a424"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:01.454138 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.454068    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8541d27-56d1-4b97-9cec-0557d180a424" (UID: "d8541d27-56d1-4b97-9cec-0557d180a424"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:01.454244 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.454226    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:01.454287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.454249    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:01.454287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.454260    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:01.455812 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.455791    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8541d27-56d1-4b97-9cec-0557d180a424-kube-api-access-9w66p" (OuterVolumeSpecName: "kube-api-access-9w66p") pod "d8541d27-56d1-4b97-9cec-0557d180a424" (UID: "d8541d27-56d1-4b97-9cec-0557d180a424"). InnerVolumeSpecName "kube-api-access-9w66p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:05:01.456169 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.456151    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-dshm" (OuterVolumeSpecName: "dshm") pod "d8541d27-56d1-4b97-9cec-0557d180a424" (UID: "d8541d27-56d1-4b97-9cec-0557d180a424"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:01.456239 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.456165    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8541d27-56d1-4b97-9cec-0557d180a424-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8541d27-56d1-4b97-9cec-0557d180a424" (UID: "d8541d27-56d1-4b97-9cec-0557d180a424"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:05:01.555012 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.554950    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8541d27-56d1-4b97-9cec-0557d180a424-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:01.555012 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.555004    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w66p\" (UniqueName: \"kubernetes.io/projected/d8541d27-56d1-4b97-9cec-0557d180a424-kube-api-access-9w66p\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:01.555012 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.555018    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8541d27-56d1-4b97-9cec-0557d180a424-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:01.851454 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:01.851351    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:05:02.259053 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.259008    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww" podUID="74854437-4fd9-48f7-8afb-801623a34297" containerName="main" containerID="cri-o://d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605" gracePeriod=45
Sep 11 20:05:02.308126 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.306933    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m" event={"ID":"d8541d27-56d1-4b97-9cec-0557d180a424","Type":"ContainerDied","Data":"76f872e85f4875e30f98a47878ee40f011a026ebbd2afbb4e49a2283fd47a0c4"}
Sep 11 20:05:02.308126 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.306993    2571 scope.go:117] "RemoveContainer" containerID="a9a98f9e324d6dbf494c03c4f47734971145fee4a43d2a0b2ab58a92f8d8a3a0"
Sep 11 20:05:02.308126 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.307187    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"
Sep 11 20:05:02.330645 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.330600    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"]
Sep 11 20:05:02.333255 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.333227    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-57d97d/rate-test-model-57d97d-kserve-856df488d7-9n89m"]
Sep 11 20:05:02.552885 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.552860    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:05:02.665370 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665333    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-tmp-dir\") pod \"74854437-4fd9-48f7-8afb-801623a34297\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") "
Sep 11 20:05:02.665370 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665372    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kznkl\" (UniqueName: \"kubernetes.io/projected/74854437-4fd9-48f7-8afb-801623a34297-kube-api-access-kznkl\") pod \"74854437-4fd9-48f7-8afb-801623a34297\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") "
Sep 11 20:05:02.665600 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665389    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-model-cache\") pod \"74854437-4fd9-48f7-8afb-801623a34297\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") "
Sep 11 20:05:02.665600 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665434    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-home\") pod \"74854437-4fd9-48f7-8afb-801623a34297\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") "
Sep 11 20:05:02.665600 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665467    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74854437-4fd9-48f7-8afb-801623a34297-tls-certs\") pod \"74854437-4fd9-48f7-8afb-801623a34297\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") "
Sep 11 20:05:02.665600 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665493    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-dshm\") pod \"74854437-4fd9-48f7-8afb-801623a34297\" (UID: \"74854437-4fd9-48f7-8afb-801623a34297\") "
Sep 11 20:05:02.665600 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665591    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74854437-4fd9-48f7-8afb-801623a34297" (UID: "74854437-4fd9-48f7-8afb-801623a34297"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:02.665835 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665678    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-home" (OuterVolumeSpecName: "home") pod "74854437-4fd9-48f7-8afb-801623a34297" (UID: "74854437-4fd9-48f7-8afb-801623a34297"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:02.665835 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665714    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-model-cache" (OuterVolumeSpecName: "model-cache") pod "74854437-4fd9-48f7-8afb-801623a34297" (UID: "74854437-4fd9-48f7-8afb-801623a34297"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:02.665835 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665739    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:02.665835 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.665751    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:02.667599 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.667575    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74854437-4fd9-48f7-8afb-801623a34297-kube-api-access-kznkl" (OuterVolumeSpecName: "kube-api-access-kznkl") pod "74854437-4fd9-48f7-8afb-801623a34297" (UID: "74854437-4fd9-48f7-8afb-801623a34297"). InnerVolumeSpecName "kube-api-access-kznkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:05:02.667599 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.667589    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-dshm" (OuterVolumeSpecName: "dshm") pod "74854437-4fd9-48f7-8afb-801623a34297" (UID: "74854437-4fd9-48f7-8afb-801623a34297"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:02.667599 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.667584    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74854437-4fd9-48f7-8afb-801623a34297-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74854437-4fd9-48f7-8afb-801623a34297" (UID: "74854437-4fd9-48f7-8afb-801623a34297"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:05:02.766327 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.766288    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:02.766327 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.766322    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74854437-4fd9-48f7-8afb-801623a34297-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:02.766327 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.766331    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74854437-4fd9-48f7-8afb-801623a34297-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:02.766589 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:02.766342    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kznkl\" (UniqueName: \"kubernetes.io/projected/74854437-4fd9-48f7-8afb-801623a34297-kube-api-access-kznkl\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:05:03.338971 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.338936    2571 generic.go:356] "Generic (PLEG): container finished" podID="74854437-4fd9-48f7-8afb-801623a34297" containerID="d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605" exitCode=0
Sep 11 20:05:03.339454 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.338981    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww" event={"ID":"74854437-4fd9-48f7-8afb-801623a34297","Type":"ContainerDied","Data":"d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605"}
Sep 11 20:05:03.339454 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.338998    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"
Sep 11 20:05:03.339454 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.339017    2571 scope.go:117] "RemoveContainer" containerID="d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605"
Sep 11 20:05:03.339454 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.339007    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww" event={"ID":"74854437-4fd9-48f7-8afb-801623a34297","Type":"ContainerDied","Data":"87ad5ec2d339c8332abaae93c817057d96761104e54648475bcd78bc3ed219f2"}
Sep 11 20:05:03.348634 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.348616    2571 scope.go:117] "RemoveContainer" containerID="d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605"
Sep 11 20:05:03.348902 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:05:03.348881    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605\": container with ID starting with d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605 not found: ID does not exist" containerID="d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605"
Sep 11 20:05:03.348940 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.348913    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605"} err="failed to get container status \"d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605\": rpc error: code = NotFound desc = could not find container \"d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605\": container with ID starting with d369e34351b6168701001c8ef2c2e2d1237bc69ce4c1edaa9d6b9d49a5a5d605 not found: ID does not exist"
Sep 11 20:05:03.360061 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.360036    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"]
Sep 11 20:05:03.364244 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.364220    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8a4eb7/rate-test-model-8a4eb7-kserve-5f86d596dc-c85ww"]
Sep 11 20:05:03.872917 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.872884    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74854437-4fd9-48f7-8afb-801623a34297" path="/var/lib/kubelet/pods/74854437-4fd9-48f7-8afb-801623a34297/volumes"
Sep 11 20:05:03.873296 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:03.873283    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8541d27-56d1-4b97-9cec-0557d180a424" path="/var/lib/kubelet/pods/d8541d27-56d1-4b97-9cec-0557d180a424/volumes"
Sep 11 20:05:06.279263 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:06.279220    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 20:05:06.279627 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:06.279291    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 20:05:07.363065 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:07.362964    2571 generic.go:356] "Generic (PLEG): container finished" podID="03884f0e-1142-4530-a781-a4dfebb30804" containerID="6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c" exitCode=0
Sep 11 20:05:07.363065 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:07.363019    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" event={"ID":"03884f0e-1142-4530-a781-a4dfebb30804","Type":"ContainerDied","Data":"6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c"}
Sep 11 20:05:07.363546 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:07.363409    2571 scope.go:117] "RemoveContainer" containerID="6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c"
Sep 11 20:05:07.364500 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:07.364471    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:07.364590 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:07.364574    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:07.364639 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:07.364611    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:08.370609 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:08.370570    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" event={"ID":"03884f0e-1142-4530-a781-a4dfebb30804","Type":"ContainerStarted","Data":"72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b"}
Sep 11 20:05:09.267804 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:09.267763    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:05:09.268033 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:09.268005    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 20:05:09.268095 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:09.268076    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 20:05:10.267759 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:10.267722    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 20:05:10.268187 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:10.267779    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 20:05:11.267458 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:11.267419    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 20:05:11.267683 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:11.267497    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 20:05:12.267271 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:12.267229    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 20:05:12.267671 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:12.267292    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 20:05:13.267564 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:13.267520    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Sep 11 20:05:13.267958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:13.267588    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Sep 11 20:05:14.317051 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:14.317017    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:05:14.317562 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:14.317447    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:05:14.318156 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:14.318135    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:05:17.047747 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.047666    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"]
Sep 11 20:05:17.048090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.048017    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74854437-4fd9-48f7-8afb-801623a34297" containerName="main"
Sep 11 20:05:17.048090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.048030    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="74854437-4fd9-48f7-8afb-801623a34297" containerName="main"
Sep 11 20:05:17.048090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.048040    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8541d27-56d1-4b97-9cec-0557d180a424" containerName="main"
Sep 11 20:05:17.048090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.048046    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8541d27-56d1-4b97-9cec-0557d180a424" containerName="main"
Sep 11 20:05:17.048268 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.048098    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8541d27-56d1-4b97-9cec-0557d180a424" containerName="main"
Sep 11 20:05:17.048268 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.048131    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="74854437-4fd9-48f7-8afb-801623a34297" containerName="main"
Sep 11 20:05:17.049914 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.049897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.054478 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.054447    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-df1a8a-openshift-default-dockercfg-k6b9q\""
Sep 11 20:05:17.080074 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.080040    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"]
Sep 11 20:05:17.196740 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196695    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adde3c4-cdb6-4b71-8606-39eb008b1469-istiod-ca-cert\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.196740 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-certs\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.196958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5rb8\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-kube-api-access-d5rb8\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.196958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-data\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.196958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196821    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-credential-socket\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.196958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196889    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-token\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.196958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196928    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-envoy\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.196958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-socket\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.197199 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.196975    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-podinfo\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298180    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-token\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298239    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-envoy\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298260    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-socket\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-podinfo\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298654 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adde3c4-cdb6-4b71-8606-39eb008b1469-istiod-ca-cert\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298654 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-certs\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298654 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298380    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5rb8\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-kube-api-access-d5rb8\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298654 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-data\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298654 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-credential-socket\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.298919 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.298738    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-socket\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.299074 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.299054    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-credential-socket\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.299159 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.299092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-certs\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.299216 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.299187    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adde3c4-cdb6-4b71-8606-39eb008b1469-istiod-ca-cert\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.299216 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.299188    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-data\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.300718 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.300698    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-envoy\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.300933 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.300914    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-podinfo\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.306443 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.306423    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-token\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.306827 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.306809    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5rb8\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-kube-api-access-d5rb8\") pod \"e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") " pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.360273 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.360243    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:17.637993 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.637959    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"]
Sep 11 20:05:17.639180 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:05:17.639150    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adde3c4_cdb6_4b71_8606_39eb008b1469.slice/crio-e9f31421375602fc68db3d60eef9920fb13a800420915a6e93d7baacbbbc23c9 WatchSource:0}: Error finding container e9f31421375602fc68db3d60eef9920fb13a800420915a6e93d7baacbbbc23c9: Status 404 returned error can't find the container with id e9f31421375602fc68db3d60eef9920fb13a800420915a6e93d7baacbbbc23c9
Sep 11 20:05:17.641382 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.641351    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:17.641447 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.641417    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:17.641484 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:17.641449    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:18.429186 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:18.429142    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" event={"ID":"9adde3c4-cdb6-4b71-8606-39eb008b1469","Type":"ContainerStarted","Data":"eee7b9c0e7463a1955126aae660acb7bc5917e5e68339eb79ebd7427b6c9b2b5"}
Sep 11 20:05:18.429627 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:18.429203    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" event={"ID":"9adde3c4-cdb6-4b71-8606-39eb008b1469","Type":"ContainerStarted","Data":"e9f31421375602fc68db3d60eef9920fb13a800420915a6e93d7baacbbbc23c9"}
Sep 11 20:05:18.451018 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:18.450967    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" podStartSLOduration=1.450950793 podStartE2EDuration="1.450950793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:18.450277825 +0000 UTC m=+2155.126214733" watchObservedRunningTime="2026-09-11 20:05:18.450950793 +0000 UTC m=+2155.126887700"
Sep 11 20:05:19.361307 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:19.361267    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:19.366313 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:19.366287    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:19.434568 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:19.434534    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:19.435388 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:19.435368    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:23.598993 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.598955    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"]
Sep 11 20:05:23.602275 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.602257    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:23.605082 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.605052    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-df1a8a\""
Sep 11 20:05:23.605247 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.605228    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-df1a8a-dockercfg-zcrg5\""
Sep 11 20:05:23.619795 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.619765    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"]
Sep 11 20:05:23.628438 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.628409    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"]
Sep 11 20:05:23.632003 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.631984    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:23.643339 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.643314    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"]
Sep 11 20:05:23.754982 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.754946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e9458a-b94f-42ab-a3d9-35784479c203-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt\" (UID: \"09e9458a-b94f-42ab-a3d9-35784479c203\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:23.755204 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.755006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsws\" (UniqueName: \"kubernetes.io/projected/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-kube-api-access-zmsws\") pod \"payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4\" (UID: \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:23.755204 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.755074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4\" (UID: \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:23.755204 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.755171    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh772\" (UniqueName: \"kubernetes.io/projected/09e9458a-b94f-42ab-a3d9-35784479c203-kube-api-access-qh772\") pod \"payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt\" (UID: \"09e9458a-b94f-42ab-a3d9-35784479c203\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:23.855897 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.855805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsws\" (UniqueName: \"kubernetes.io/projected/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-kube-api-access-zmsws\") pod \"payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4\" (UID: \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:23.855897 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.855842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4\" (UID: \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:23.855897 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.855895    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh772\" (UniqueName: \"kubernetes.io/projected/09e9458a-b94f-42ab-a3d9-35784479c203-kube-api-access-qh772\") pod \"payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt\" (UID: \"09e9458a-b94f-42ab-a3d9-35784479c203\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:23.856293 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.855957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e9458a-b94f-42ab-a3d9-35784479c203-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt\" (UID: \"09e9458a-b94f-42ab-a3d9-35784479c203\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:23.863327 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.863301    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-df1a8a\""
Sep 11 20:05:23.866800 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.866770    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4\" (UID: \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:23.866800 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.866782    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e9458a-b94f-42ab-a3d9-35784479c203-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt\" (UID: \"09e9458a-b94f-42ab-a3d9-35784479c203\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:23.867660 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.867635    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsws\" (UniqueName: \"kubernetes.io/projected/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-kube-api-access-zmsws\") pod \"payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4\" (UID: \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:23.869225 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.869200    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh772\" (UniqueName: \"kubernetes.io/projected/09e9458a-b94f-42ab-a3d9-35784479c203-kube-api-access-qh772\") pod \"payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt\" (UID: \"09e9458a-b94f-42ab-a3d9-35784479c203\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:23.914491 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.914452    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-df1a8a-dockercfg-zcrg5\""
Sep 11 20:05:23.921903 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.921843    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:23.941995 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:23.941963    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:24.203355 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.203330    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"]
Sep 11 20:05:24.203981 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:05:24.203955    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afb5f9a_b72c_4b88_be69_0e6ea6723ca8.slice/crio-9ed54c3d61d4f0edb71361cd8a9870e0577ceb33cdde94d644a2a2455154ee3e WatchSource:0}: Error finding container 9ed54c3d61d4f0edb71361cd8a9870e0577ceb33cdde94d644a2a2455154ee3e: Status 404 returned error can't find the container with id 9ed54c3d61d4f0edb71361cd8a9870e0577ceb33cdde94d644a2a2455154ee3e
Sep 11 20:05:24.231389 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.231366    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"]
Sep 11 20:05:24.233481 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:05:24.233457    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e9458a_b94f_42ab_a3d9_35784479c203.slice/crio-f73bdbc1829f4bc92c99e2d2a4bd9c808e9ccb5fbe47934cca1238e855e1cc40 WatchSource:0}: Error finding container f73bdbc1829f4bc92c99e2d2a4bd9c808e9ccb5fbe47934cca1238e855e1cc40: Status 404 returned error can't find the container with id f73bdbc1829f4bc92c99e2d2a4bd9c808e9ccb5fbe47934cca1238e855e1cc40
Sep 11 20:05:24.465042 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.464944    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt" event={"ID":"09e9458a-b94f-42ab-a3d9-35784479c203","Type":"ContainerStarted","Data":"26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407"}
Sep 11 20:05:24.465042 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.465006    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt" event={"ID":"09e9458a-b94f-42ab-a3d9-35784479c203","Type":"ContainerStarted","Data":"f73bdbc1829f4bc92c99e2d2a4bd9c808e9ccb5fbe47934cca1238e855e1cc40"}
Sep 11 20:05:24.465042 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.465036    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:24.466578 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.466545    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4" event={"ID":"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8","Type":"ContainerStarted","Data":"c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8"}
Sep 11 20:05:24.466697 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.466603    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4" event={"ID":"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8","Type":"ContainerStarted","Data":"9ed54c3d61d4f0edb71361cd8a9870e0577ceb33cdde94d644a2a2455154ee3e"}
Sep 11 20:05:24.466697 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.466631    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:24.490007 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.489953    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt" podStartSLOduration=1.4899368069999999 podStartE2EDuration="1.489936807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:24.488188384 +0000 UTC m=+2161.164125294" watchObservedRunningTime="2026-09-11 20:05:24.489936807 +0000 UTC m=+2161.165873709"
Sep 11 20:05:24.524695 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:24.524636    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4" podStartSLOduration=1.524615997 podStartE2EDuration="1.524615997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:24.523066981 +0000 UTC m=+2161.199003882" watchObservedRunningTime="2026-09-11 20:05:24.524615997 +0000 UTC m=+2161.200552904"
Sep 11 20:05:27.849978 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:27.849937    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:05:35.474762 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:35.474727    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:05:35.475283 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:35.474817    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:05:38.554463 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:38.554424    2571 generic.go:356] "Generic (PLEG): container finished" podID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerID="eee7b9c0e7463a1955126aae660acb7bc5917e5e68339eb79ebd7427b6c9b2b5" exitCode=0
Sep 11 20:05:38.554952 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:38.554501    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" event={"ID":"9adde3c4-cdb6-4b71-8606-39eb008b1469","Type":"ContainerDied","Data":"eee7b9c0e7463a1955126aae660acb7bc5917e5e68339eb79ebd7427b6c9b2b5"}
Sep 11 20:05:38.554952 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:38.554891    2571 scope.go:117] "RemoveContainer" containerID="eee7b9c0e7463a1955126aae660acb7bc5917e5e68339eb79ebd7427b6c9b2b5"
Sep 11 20:05:38.556037 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:38.556006    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:38.556159 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:38.556073    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:38.556159 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:38.556129    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 20:05:39.563522 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:39.563486    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" event={"ID":"9adde3c4-cdb6-4b71-8606-39eb008b1469","Type":"ContainerStarted","Data":"7be325ebea0cb2262a84209e91dcdf81dc2080ec64bdf7efb90b07caa31bb625"}
Sep 11 20:05:40.361212 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:40.361163    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:40.361412 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:40.361381    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 20:05:40.361459 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:40.361432    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 20:05:41.361162 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:41.361125    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 20:05:41.361555 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:41.361183    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 20:05:42.361337 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:42.361295    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 20:05:42.361726 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:42.361362    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 20:05:43.361060 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:43.361012    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 20:05:43.361282 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:43.361092    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 20:05:44.361292 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:44.361246    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 11 20:05:44.361769 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:44.361312    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 11 20:05:45.365048 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:45.365016    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:45.365651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:45.365325    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:45.366122 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:45.366085    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:05:50.332618 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:50.332528    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt_09e9458a-b94f-42ab-a3d9-35784479c203/payload-processing/0.log"
Sep 11 20:05:57.436698 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.436658    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"]
Sep 11 20:05:57.441152 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.441117    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.443887 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.443858    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-96e341\"/\"kube-root-ca.crt\""
Sep 11 20:05:57.445280 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.445250    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-96e341\"/\"openshift-service-ca.crt\""
Sep 11 20:05:57.445280 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.445255    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-96e341\"/\"default-dockercfg-rnm6l\""
Sep 11 20:05:57.445448 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.445361    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-96e341\"/\"ipp-route-96e341-kserve-self-signed-certs\""
Sep 11 20:05:57.451080 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.451060    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"]
Sep 11 20:05:57.579908 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.579845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tls-certs\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.580183 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.579938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-model-cache\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.580183 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.580032    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76t5\" (UniqueName: \"kubernetes.io/projected/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-kube-api-access-f76t5\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.580183 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.580061    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-home\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.580183 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.580133    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tmp-dir\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.580183 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.580158    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-dshm\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.680925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.680887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tmp-dir\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.680925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.680924    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-dshm\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.681188 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.680956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tls-certs\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.681188 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.681012    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-model-cache\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.681188 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.681085    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76t5\" (UniqueName: \"kubernetes.io/projected/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-kube-api-access-f76t5\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.681188 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.681140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-home\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.681390 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.681371    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tmp-dir\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.681454 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.681416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-home\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.681509 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.681479    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-model-cache\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.683299 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.683277    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-dshm\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.683619 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.683599    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tls-certs\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.689629 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.689558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76t5\" (UniqueName: \"kubernetes.io/projected/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-kube-api-access-f76t5\") pod \"ipp-route-96e341-kserve-6cc76fd75-xwlqg\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") " pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:57.751573 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:57.751533    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:05:58.025704 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:58.025679    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"]
Sep 11 20:05:58.027454 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:05:58.027420    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b59bab0_8fdc_4f55_98c1_9c9860ab1cdf.slice/crio-bd22e9dabc4371cd1c48734dc0da9505e31c43710fdc95e3a8a96a9f4f3e702f WatchSource:0}: Error finding container bd22e9dabc4371cd1c48734dc0da9505e31c43710fdc95e3a8a96a9f4f3e702f: Status 404 returned error can't find the container with id bd22e9dabc4371cd1c48734dc0da9505e31c43710fdc95e3a8a96a9f4f3e702f
Sep 11 20:05:58.674472 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:58.674434    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg" event={"ID":"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf","Type":"ContainerStarted","Data":"0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b"}
Sep 11 20:05:58.674472 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:58.674481    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg" event={"ID":"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf","Type":"ContainerStarted","Data":"bd22e9dabc4371cd1c48734dc0da9505e31c43710fdc95e3a8a96a9f4f3e702f"}
Sep 11 20:05:58.696135 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:05:58.696070    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg" podStartSLOduration=1.696052319 podStartE2EDuration="1.696052319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:58.694118774 +0000 UTC m=+2195.370055679" watchObservedRunningTime="2026-09-11 20:05:58.696052319 +0000 UTC m=+2195.371989227"
Sep 11 20:06:01.263695 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:01.263659    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:06:07.752538 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:07.752501    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:06:07.753027 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:07.752561    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:06:07.765444 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:07.765417    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:06:08.739653 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:08.739612    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:06:14.556499 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:14.556459    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:06:21.103378 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:21.103346    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt_09e9458a-b94f-42ab-a3d9-35784479c203/payload-processing/0.log"
Sep 11 20:06:24.190505 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:24.190471    2571 scope.go:117] "RemoveContainer" containerID="47a6fbd0c6fc0889d23f2cb9f3b08369cac98b50e58dcf0585252e0bae005d7c"
Sep 11 20:06:24.199182 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:24.199160    2571 scope.go:117] "RemoveContainer" containerID="ec4e6a826cd8ca2e59c1f93fdf0862da1038c203987e428a566d85d754494793"
Sep 11 20:06:24.207482 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:24.207462    2571 scope.go:117] "RemoveContainer" containerID="bb29c1c61dcc2f2e6a013f802258a5792b7fbf0f5fe9fcdecc6ce4271e1809c6"
Sep 11 20:06:24.215093 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:24.215069    2571 scope.go:117] "RemoveContainer" containerID="f59f8ca35187edd26a61d607c1a67306e20af7d6f1b12e945c5f6af88aef74d9"
Sep 11 20:06:29.334486 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.334458    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"]
Sep 11 20:06:29.337730 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.337712    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:29.340701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.340682    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2f8fd9\""
Sep 11 20:06:29.340797 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.340733    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2f8fd9-dockercfg-xns7m\""
Sep 11 20:06:29.349043 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.349022    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"]
Sep 11 20:06:29.473024 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.472972    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52\" (UID: \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:29.473230 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.473052    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7sx\" (UniqueName: \"kubernetes.io/projected/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-kube-api-access-ft7sx\") pod \"payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52\" (UID: \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:29.574541 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.574504    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7sx\" (UniqueName: \"kubernetes.io/projected/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-kube-api-access-ft7sx\") pod \"payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52\" (UID: \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:29.574741 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.574599    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52\" (UID: \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:29.575336 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.575310    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52\" (UID: \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:29.582992 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.582968    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7sx\" (UniqueName: \"kubernetes.io/projected/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-kube-api-access-ft7sx\") pod \"payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52\" (UID: \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:29.648175 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.648142    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:29.923770 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:29.923685    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"]
Sep 11 20:06:29.926963 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:06:29.926935    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2354e4b7_4bb0_4d60_ae74_6dbe5475df15.slice/crio-d747efcbe1417e08f3a456aa9bfa185371ab55eb608115bf24840a16e88064cf WatchSource:0}: Error finding container d747efcbe1417e08f3a456aa9bfa185371ab55eb608115bf24840a16e88064cf: Status 404 returned error can't find the container with id d747efcbe1417e08f3a456aa9bfa185371ab55eb608115bf24840a16e88064cf
Sep 11 20:06:30.851784 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:30.851749    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52" event={"ID":"2354e4b7-4bb0-4d60-ae74-6dbe5475df15","Type":"ContainerStarted","Data":"6de334be72db6bfc25c57f7c316491766174d1b09f44f574113d7f78d5d019bb"}
Sep 11 20:06:30.852182 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:30.851801    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52" event={"ID":"2354e4b7-4bb0-4d60-ae74-6dbe5475df15","Type":"ContainerStarted","Data":"d747efcbe1417e08f3a456aa9bfa185371ab55eb608115bf24840a16e88064cf"}
Sep 11 20:06:30.852182 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:30.851828    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:06:30.871474 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:30.871424    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52" podStartSLOduration=1.871408698 podStartE2EDuration="1.871408698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:30.86863563 +0000 UTC m=+2227.544572559" watchObservedRunningTime="2026-09-11 20:06:30.871408698 +0000 UTC m=+2227.547345606"
Sep 11 20:06:34.157499 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:34.157458    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:06:41.859432 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:06:41.859402    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:07:06.835951 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:06.835903    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"]
Sep 11 20:07:06.836603 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:06.836148    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52" podUID="2354e4b7-4bb0-4d60-ae74-6dbe5475df15" containerName="payload-processing" containerID="cri-o://6de334be72db6bfc25c57f7c316491766174d1b09f44f574113d7f78d5d019bb" gracePeriod=30
Sep 11 20:07:07.050118 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.044131    2571 generic.go:356] "Generic (PLEG): container finished" podID="2354e4b7-4bb0-4d60-ae74-6dbe5475df15" containerID="6de334be72db6bfc25c57f7c316491766174d1b09f44f574113d7f78d5d019bb" exitCode=0
Sep 11 20:07:07.050118 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.044240    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52" event={"ID":"2354e4b7-4bb0-4d60-ae74-6dbe5475df15","Type":"ContainerDied","Data":"6de334be72db6bfc25c57f7c316491766174d1b09f44f574113d7f78d5d019bb"}
Sep 11 20:07:07.089876 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.089815    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:07:07.097088 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.097068    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft7sx\" (UniqueName: \"kubernetes.io/projected/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-kube-api-access-ft7sx\") pod \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\" (UID: \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\") "
Sep 11 20:07:07.097205 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.097141    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-plugins-config-volume\") pod \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\" (UID: \"2354e4b7-4bb0-4d60-ae74-6dbe5475df15\") "
Sep 11 20:07:07.097485 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.097463    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2354e4b7-4bb0-4d60-ae74-6dbe5475df15" (UID: "2354e4b7-4bb0-4d60-ae74-6dbe5475df15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:07.099084 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.099061    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-kube-api-access-ft7sx" (OuterVolumeSpecName: "kube-api-access-ft7sx") pod "2354e4b7-4bb0-4d60-ae74-6dbe5475df15" (UID: "2354e4b7-4bb0-4d60-ae74-6dbe5475df15"). InnerVolumeSpecName "kube-api-access-ft7sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:07.198729 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.198684    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft7sx\" (UniqueName: \"kubernetes.io/projected/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-kube-api-access-ft7sx\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:07.198729 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:07.198722    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2354e4b7-4bb0-4d60-ae74-6dbe5475df15-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:08.050963 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:08.050928    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"
Sep 11 20:07:08.051419 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:08.050960    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52" event={"ID":"2354e4b7-4bb0-4d60-ae74-6dbe5475df15","Type":"ContainerDied","Data":"d747efcbe1417e08f3a456aa9bfa185371ab55eb608115bf24840a16e88064cf"}
Sep 11 20:07:08.051419 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:08.051023    2571 scope.go:117] "RemoveContainer" containerID="6de334be72db6bfc25c57f7c316491766174d1b09f44f574113d7f78d5d019bb"
Sep 11 20:07:08.067491 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:08.067463    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"]
Sep 11 20:07:08.071183 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:08.071158    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2f8fd9-57b6bb46c7-4sc52"]
Sep 11 20:07:09.873209 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:09.873178    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2354e4b7-4bb0-4d60-ae74-6dbe5475df15" path="/var/lib/kubelet/pods/2354e4b7-4bb0-4d60-ae74-6dbe5475df15/volumes"
Sep 11 20:07:11.151977 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:11.151936    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:07:17.248632 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.248590    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"]
Sep 11 20:07:17.249025 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.248914    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2354e4b7-4bb0-4d60-ae74-6dbe5475df15" containerName="payload-processing"
Sep 11 20:07:17.249025 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.248925    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2354e4b7-4bb0-4d60-ae74-6dbe5475df15" containerName="payload-processing"
Sep 11 20:07:17.249025 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.248994    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2354e4b7-4bb0-4d60-ae74-6dbe5475df15" containerName="payload-processing"
Sep 11 20:07:17.253386 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.253368    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"
Sep 11 20:07:17.256150 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.256120    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:07:17.256284 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.256149    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:07:17.261025 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.260727    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"]
Sep 11 20:07:17.284698 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.284661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13e7c19c-1715-4ce3-9d33-dd5699131627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc\" (UID: \"13e7c19c-1715-4ce3-9d33-dd5699131627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"
Sep 11 20:07:17.385843 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.385799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13e7c19c-1715-4ce3-9d33-dd5699131627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc\" (UID: \"13e7c19c-1715-4ce3-9d33-dd5699131627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"
Sep 11 20:07:17.386491 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.386470    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13e7c19c-1715-4ce3-9d33-dd5699131627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc\" (UID: \"13e7c19c-1715-4ce3-9d33-dd5699131627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"
Sep 11 20:07:17.565033 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.564948    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"
Sep 11 20:07:17.832951 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:17.832923    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"]
Sep 11 20:07:17.834610 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:07:17.834579    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e7c19c_1715_4ce3_9d33_dd5699131627.slice/crio-c702706ebd277db16db30620cc7d23daac66095aa97b47ad04bd20a196e0ae42 WatchSource:0}: Error finding container c702706ebd277db16db30620cc7d23daac66095aa97b47ad04bd20a196e0ae42: Status 404 returned error can't find the container with id c702706ebd277db16db30620cc7d23daac66095aa97b47ad04bd20a196e0ae42
Sep 11 20:07:18.111557 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:18.111520    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc" event={"ID":"13e7c19c-1715-4ce3-9d33-dd5699131627","Type":"ContainerStarted","Data":"c702706ebd277db16db30620cc7d23daac66095aa97b47ad04bd20a196e0ae42"}
Sep 11 20:07:19.118935 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:19.118903    2571 generic.go:356] "Generic (PLEG): container finished" podID="13e7c19c-1715-4ce3-9d33-dd5699131627" containerID="a70054caf7be6c773141297e4e2ae4e4bd81f003cb0568adca961447fc0dee82" exitCode=0
Sep 11 20:07:19.119331 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:19.118978    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc" event={"ID":"13e7c19c-1715-4ce3-9d33-dd5699131627","Type":"ContainerDied","Data":"a70054caf7be6c773141297e4e2ae4e4bd81f003cb0568adca961447fc0dee82"}
Sep 11 20:07:20.253313 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:20.253285    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"
Sep 11 20:07:20.312131 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:20.312085    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13e7c19c-1715-4ce3-9d33-dd5699131627-maas-api-service-ca\") pod \"13e7c19c-1715-4ce3-9d33-dd5699131627\" (UID: \"13e7c19c-1715-4ce3-9d33-dd5699131627\") "
Sep 11 20:07:20.312441 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:20.312419    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e7c19c-1715-4ce3-9d33-dd5699131627-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13e7c19c-1715-4ce3-9d33-dd5699131627" (UID: "13e7c19c-1715-4ce3-9d33-dd5699131627"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:20.413164 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:20.413056    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13e7c19c-1715-4ce3-9d33-dd5699131627-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:21.131736 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:21.131704    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"
Sep 11 20:07:21.131917 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:21.131733    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc" event={"ID":"13e7c19c-1715-4ce3-9d33-dd5699131627","Type":"ContainerDied","Data":"c702706ebd277db16db30620cc7d23daac66095aa97b47ad04bd20a196e0ae42"}
Sep 11 20:07:21.131917 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:21.131795    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c702706ebd277db16db30620cc7d23daac66095aa97b47ad04bd20a196e0ae42"
Sep 11 20:07:24.245136 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:24.245090    2571 scope.go:117] "RemoveContainer" containerID="25c52bde824049a330f5d835f5553c9bfe8452879939d800bdeeed60def00f0a"
Sep 11 20:07:24.253217 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:24.253190    2571 scope.go:117] "RemoveContainer" containerID="9c23b4b964cb8a03a9a85bc8a2b1d205201bdfa1d72cdaea0aa99ebe4b2e0b21"
Sep 11 20:07:24.260522 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:24.260496    2571 scope.go:117] "RemoveContainer" containerID="411905526ec888ac42710022c31c6b80646450cf4c6e6e97451f4ddc4120bec9"
Sep 11 20:07:30.346916 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:30.346873    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:07:32.876666 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:32.876624    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"]
Sep 11 20:07:32.883877 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:32.883843    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-96e341-6mchc"]
Sep 11 20:07:33.874178 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:33.874144    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e7c19c-1715-4ce3-9d33-dd5699131627" path="/var/lib/kubelet/pods/13e7c19c-1715-4ce3-9d33-dd5699131627/volumes"
Sep 11 20:07:38.202717 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:38.202683    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"]
Sep 11 20:07:43.046354 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:43.046318    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:07:53.222238 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.222175    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg" podUID="3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" containerName="main" containerID="cri-o://0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b" gracePeriod=45
Sep 11 20:07:53.464417 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.464391    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:07:53.593708 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.593616    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tmp-dir\") pod \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") "
Sep 11 20:07:53.593708 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.593655    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tls-certs\") pod \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") "
Sep 11 20:07:53.593708 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.593677    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-dshm\") pod \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") "
Sep 11 20:07:53.593997 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.593719    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f76t5\" (UniqueName: \"kubernetes.io/projected/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-kube-api-access-f76t5\") pod \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") "
Sep 11 20:07:53.593997 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.593816    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-model-cache\") pod \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") "
Sep 11 20:07:53.593997 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.593840    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-home\") pod \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\" (UID: \"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf\") "
Sep 11 20:07:53.593997 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.593937    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" (UID: "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:53.594218 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.594128    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:53.594218 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.594155    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" (UID: "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:53.594218 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.594178    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-home" (OuterVolumeSpecName: "home") pod "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" (UID: "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:53.595870 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.595840    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-dshm" (OuterVolumeSpecName: "dshm") pod "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" (UID: "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:53.595994 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.595912    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" (UID: "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:07:53.595994 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.595937    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-kube-api-access-f76t5" (OuterVolumeSpecName: "kube-api-access-f76t5") pod "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" (UID: "3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf"). InnerVolumeSpecName "kube-api-access-f76t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:53.694784 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.694748    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:53.694784 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.694779    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:53.694784 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.694791    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:53.695008 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.694802    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:53.695008 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:53.694814    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f76t5\" (UniqueName: \"kubernetes.io/projected/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf-kube-api-access-f76t5\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:07:54.306690 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.306598    2571 generic.go:356] "Generic (PLEG): container finished" podID="3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" containerID="0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b" exitCode=0
Sep 11 20:07:54.306690 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.306641    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"
Sep 11 20:07:54.306690 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.306676    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg" event={"ID":"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf","Type":"ContainerDied","Data":"0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b"}
Sep 11 20:07:54.307281 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.306725    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg" event={"ID":"3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf","Type":"ContainerDied","Data":"bd22e9dabc4371cd1c48734dc0da9505e31c43710fdc95e3a8a96a9f4f3e702f"}
Sep 11 20:07:54.307281 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.306743    2571 scope.go:117] "RemoveContainer" containerID="0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b"
Sep 11 20:07:54.315306 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.315289    2571 scope.go:117] "RemoveContainer" containerID="0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b"
Sep 11 20:07:54.315576 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:07:54.315558    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b\": container with ID starting with 0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b not found: ID does not exist" containerID="0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b"
Sep 11 20:07:54.315628 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.315585    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b"} err="failed to get container status \"0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b\": rpc error: code = NotFound desc = could not find container \"0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b\": container with ID starting with 0ce6b0d9f03091f6087a02bbefb9fe047df12d52f010590a28fa5d8e7e597f6b not found: ID does not exist"
Sep 11 20:07:54.323478 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.323456    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"]
Sep 11 20:07:54.325521 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:54.325499    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-96e341/ipp-route-96e341-kserve-6cc76fd75-xwlqg"]
Sep 11 20:07:55.873032 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:07:55.873000    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" path="/var/lib/kubelet/pods/3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf/volumes"
Sep 11 20:08:21.707037 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:21.707002    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"]
Sep 11 20:08:21.707605 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:21.707276    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy" containerID="cri-o://72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b" gracePeriod=30
Sep 11 20:08:24.288674 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:24.288636    2571 scope.go:117] "RemoveContainer" containerID="949a61888b276621a9b0d7cc59d929b90be25dfca21910868992c989e8957b30"
Sep 11 20:08:24.297348 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:24.297327    2571 scope.go:117] "RemoveContainer" containerID="eff199182b4abf4bbcd3311ddc9280e721003cd8997b48c3ec4de1af294a1b01"
Sep 11 20:08:26.954651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:26.954626    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:08:27.062420 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062333    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-certs\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062420 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062369    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-data\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062420 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062407    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-socket\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062444    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-istio-token\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062480    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884f0e-1142-4530-a781-a4dfebb30804-istiod-ca-cert\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062510    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-envoy\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062555    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmk8k\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-kube-api-access-wmk8k\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062588    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-credential-socket\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062648    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884f0e-1142-4530-a781-a4dfebb30804-istio-podinfo\") pod \"03884f0e-1142-4530-a781-a4dfebb30804\" (UID: \"03884f0e-1142-4530-a781-a4dfebb30804\") "
Sep 11 20:08:27.062701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062652    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:27.062701 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062664    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-data" (OuterVolumeSpecName: "istio-data") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:27.063068 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062802    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:27.063068 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062909    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:27.063068 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.062984    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.063068 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.063006    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.063068 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.063021    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.063068 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.063034    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.063344 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.063146    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03884f0e-1142-4530-a781-a4dfebb30804-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:08:27.064745 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.064721    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-istio-token" (OuterVolumeSpecName: "istio-token") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:08:27.064957 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.064937    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:27.065074 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.064969    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-kube-api-access-wmk8k" (OuterVolumeSpecName: "kube-api-access-wmk8k") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "kube-api-access-wmk8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:08:27.065144 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.065060    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03884f0e-1142-4530-a781-a4dfebb30804-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03884f0e-1142-4530-a781-a4dfebb30804" (UID: "03884f0e-1142-4530-a781-a4dfebb30804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 20:08:27.163781 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.163745    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03884f0e-1142-4530-a781-a4dfebb30804-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.163781 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.163783    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmk8k\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-kube-api-access-wmk8k\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.163990 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.163799    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03884f0e-1142-4530-a781-a4dfebb30804-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.163990 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.163812    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03884f0e-1142-4530-a781-a4dfebb30804-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.163990 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.163824    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03884f0e-1142-4530-a781-a4dfebb30804-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:27.479031 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.478996    2571 generic.go:356] "Generic (PLEG): container finished" podID="03884f0e-1142-4530-a781-a4dfebb30804" containerID="72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b" exitCode=0
Sep 11 20:08:27.479220 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.479043    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" event={"ID":"03884f0e-1142-4530-a781-a4dfebb30804","Type":"ContainerDied","Data":"72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b"}
Sep 11 20:08:27.479220 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.479049    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"
Sep 11 20:08:27.479220 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.479096    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4" event={"ID":"03884f0e-1142-4530-a781-a4dfebb30804","Type":"ContainerDied","Data":"05a43d308293e85d285dbb71465928e0c190827dc5dcc43ee35e6b1dd9a2d88b"}
Sep 11 20:08:27.479220 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.479142    2571 scope.go:117] "RemoveContainer" containerID="72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b"
Sep 11 20:08:27.488483 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.488456    2571 scope.go:117] "RemoveContainer" containerID="6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c"
Sep 11 20:08:27.496960 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.496941    2571 scope.go:117] "RemoveContainer" containerID="72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b"
Sep 11 20:08:27.497246 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:08:27.497224    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b\": container with ID starting with 72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b not found: ID does not exist" containerID="72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b"
Sep 11 20:08:27.497322 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.497258    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b"} err="failed to get container status \"72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b\": rpc error: code = NotFound desc = could not find container \"72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b\": container with ID starting with 72c5b000110bab2f99186da3ba3c196ba2e845484bb59b170cb852c1fdcc087b not found: ID does not exist"
Sep 11 20:08:27.497322 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.497282    2571 scope.go:117] "RemoveContainer" containerID="6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c"
Sep 11 20:08:27.497527 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:08:27.497505    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c\": container with ID starting with 6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c not found: ID does not exist" containerID="6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c"
Sep 11 20:08:27.497567 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.497533    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c"} err="failed to get container status \"6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c\": rpc error: code = NotFound desc = could not find container \"6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c\": container with ID starting with 6ddfa8b72bbcba6ee8c3011b44cdb25b5e080240c7ed0844c18f6957c881818c not found: ID does not exist"
Sep 11 20:08:27.502328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.502306    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"]
Sep 11 20:08:27.507419 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.507399    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-96e341-openshift-default-6b66cd7c8-99bs4"]
Sep 11 20:08:27.873003 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:27.872975    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03884f0e-1142-4530-a781-a4dfebb30804" path="/var/lib/kubelet/pods/03884f0e-1142-4530-a781-a4dfebb30804/volumes"
Sep 11 20:08:32.681931 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:32.681893    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"]
Sep 11 20:08:32.682405 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:32.682111    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt" podUID="09e9458a-b94f-42ab-a3d9-35784479c203" containerName="payload-processing" containerID="cri-o://26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407" gracePeriod=30
Sep 11 20:08:32.691242 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:32.690713    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"]
Sep 11 20:08:32.691242 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:32.690947    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4" podUID="5afb5f9a-b72c-4b88-be69-0e6ea6723ca8" containerName="payload-pre-processing" containerID="cri-o://c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8" gracePeriod=30
Sep 11 20:08:32.954042 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:32.954021    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:08:32.956403 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:32.956384    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:08:33.011788 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.011755    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh772\" (UniqueName: \"kubernetes.io/projected/09e9458a-b94f-42ab-a3d9-35784479c203-kube-api-access-qh772\") pod \"09e9458a-b94f-42ab-a3d9-35784479c203\" (UID: \"09e9458a-b94f-42ab-a3d9-35784479c203\") "
Sep 11 20:08:33.011925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.011817    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-plugins-config-volume\") pod \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\" (UID: \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\") "
Sep 11 20:08:33.011925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.011861    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmsws\" (UniqueName: \"kubernetes.io/projected/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-kube-api-access-zmsws\") pod \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\" (UID: \"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8\") "
Sep 11 20:08:33.011925 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.011919    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e9458a-b94f-42ab-a3d9-35784479c203-plugins-config-volume\") pod \"09e9458a-b94f-42ab-a3d9-35784479c203\" (UID: \"09e9458a-b94f-42ab-a3d9-35784479c203\") "
Sep 11 20:08:33.012289 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.012261    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5afb5f9a-b72c-4b88-be69-0e6ea6723ca8" (UID: "5afb5f9a-b72c-4b88-be69-0e6ea6723ca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:08:33.012404 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.012383    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e9458a-b94f-42ab-a3d9-35784479c203-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09e9458a-b94f-42ab-a3d9-35784479c203" (UID: "09e9458a-b94f-42ab-a3d9-35784479c203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:08:33.013896 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.013874    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e9458a-b94f-42ab-a3d9-35784479c203-kube-api-access-qh772" (OuterVolumeSpecName: "kube-api-access-qh772") pod "09e9458a-b94f-42ab-a3d9-35784479c203" (UID: "09e9458a-b94f-42ab-a3d9-35784479c203"). InnerVolumeSpecName "kube-api-access-qh772". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:08:33.013972 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.013906    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-kube-api-access-zmsws" (OuterVolumeSpecName: "kube-api-access-zmsws") pod "5afb5f9a-b72c-4b88-be69-0e6ea6723ca8" (UID: "5afb5f9a-b72c-4b88-be69-0e6ea6723ca8"). InnerVolumeSpecName "kube-api-access-zmsws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:08:33.113364 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.113328    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh772\" (UniqueName: \"kubernetes.io/projected/09e9458a-b94f-42ab-a3d9-35784479c203-kube-api-access-qh772\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:33.113364 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.113356    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:33.113364 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.113365    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmsws\" (UniqueName: \"kubernetes.io/projected/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8-kube-api-access-zmsws\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:33.113364 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.113373    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e9458a-b94f-42ab-a3d9-35784479c203-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:33.514254 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.514219    2571 generic.go:356] "Generic (PLEG): container finished" podID="09e9458a-b94f-42ab-a3d9-35784479c203" containerID="26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407" exitCode=0
Sep 11 20:08:33.514416 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.514269    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"
Sep 11 20:08:33.514416 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.514306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt" event={"ID":"09e9458a-b94f-42ab-a3d9-35784479c203","Type":"ContainerDied","Data":"26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407"}
Sep 11 20:08:33.514416 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.514357    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt" event={"ID":"09e9458a-b94f-42ab-a3d9-35784479c203","Type":"ContainerDied","Data":"f73bdbc1829f4bc92c99e2d2a4bd9c808e9ccb5fbe47934cca1238e855e1cc40"}
Sep 11 20:08:33.514416 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.514386    2571 scope.go:117] "RemoveContainer" containerID="26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407"
Sep 11 20:08:33.515674 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.515647    2571 generic.go:356] "Generic (PLEG): container finished" podID="5afb5f9a-b72c-4b88-be69-0e6ea6723ca8" containerID="c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8" exitCode=0
Sep 11 20:08:33.515790 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.515677    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4" event={"ID":"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8","Type":"ContainerDied","Data":"c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8"}
Sep 11 20:08:33.515790 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.515701    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"
Sep 11 20:08:33.515790 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.515723    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4" event={"ID":"5afb5f9a-b72c-4b88-be69-0e6ea6723ca8","Type":"ContainerDied","Data":"9ed54c3d61d4f0edb71361cd8a9870e0577ceb33cdde94d644a2a2455154ee3e"}
Sep 11 20:08:33.523487 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.523469    2571 scope.go:117] "RemoveContainer" containerID="26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407"
Sep 11 20:08:33.523782 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:08:33.523761    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407\": container with ID starting with 26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407 not found: ID does not exist" containerID="26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407"
Sep 11 20:08:33.523906 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.523799    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407"} err="failed to get container status \"26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407\": rpc error: code = NotFound desc = could not find container \"26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407\": container with ID starting with 26f3983b0fc28310756f77f8dfb14d5a9b874a9042138e0c775fd794d7e3f407 not found: ID does not exist"
Sep 11 20:08:33.523906 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.523824    2571 scope.go:117] "RemoveContainer" containerID="c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8"
Sep 11 20:08:33.532330 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.532315    2571 scope.go:117] "RemoveContainer" containerID="c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8"
Sep 11 20:08:33.532596 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:08:33.532579    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8\": container with ID starting with c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8 not found: ID does not exist" containerID="c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8"
Sep 11 20:08:33.532647 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.532602    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8"} err="failed to get container status \"c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8\": rpc error: code = NotFound desc = could not find container \"c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8\": container with ID starting with c90f1fae3812b61c1b1bc97c4a72993e62a651235bf4fae3e34610b15ca0c2c8 not found: ID does not exist"
Sep 11 20:08:33.551848 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.551828    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"]
Sep 11 20:08:33.570884 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.570865    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df1a8a-7dd6c78b85-ldznt"]
Sep 11 20:08:33.587428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.587405    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"]
Sep 11 20:08:33.591892 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.591873    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df1a8a-5c5d4545d6-c6fz4"]
Sep 11 20:08:33.872332 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.872306    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e9458a-b94f-42ab-a3d9-35784479c203" path="/var/lib/kubelet/pods/09e9458a-b94f-42ab-a3d9-35784479c203/volumes"
Sep 11 20:08:33.872662 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:33.872635    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5afb5f9a-b72c-4b88-be69-0e6ea6723ca8" path="/var/lib/kubelet/pods/5afb5f9a-b72c-4b88-be69-0e6ea6723ca8/volumes"
Sep 11 20:08:36.158421 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:36.158383    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:08:39.415143 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.415081    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"]
Sep 11 20:08:39.415621 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.415361    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy" containerID="cri-o://7be325ebea0cb2262a84209e91dcdf81dc2080ec64bdf7efb90b07caa31bb625" gracePeriod=30
Sep 11 20:08:39.788060 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.787968    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"]
Sep 11 20:08:39.788287 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788272    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy"
Sep 11 20:08:39.788328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788288    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy"
Sep 11 20:08:39.788328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788299    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e9458a-b94f-42ab-a3d9-35784479c203" containerName="payload-processing"
Sep 11 20:08:39.788328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788305    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e9458a-b94f-42ab-a3d9-35784479c203" containerName="payload-processing"
Sep 11 20:08:39.788328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788322    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afb5f9a-b72c-4b88-be69-0e6ea6723ca8" containerName="payload-pre-processing"
Sep 11 20:08:39.788328 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788329    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afb5f9a-b72c-4b88-be69-0e6ea6723ca8" containerName="payload-pre-processing"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788335    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e7c19c-1715-4ce3-9d33-dd5699131627" containerName="revoke-keys"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788340    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e7c19c-1715-4ce3-9d33-dd5699131627" containerName="revoke-keys"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788354    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" containerName="main"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788359    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" containerName="main"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788403    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e9458a-b94f-42ab-a3d9-35784479c203" containerName="payload-processing"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788410    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788415    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e7c19c-1715-4ce3-9d33-dd5699131627" containerName="revoke-keys"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788422    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b59bab0-8fdc-4f55-98c1-9c9860ab1cdf" containerName="main"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788429    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy"
Sep 11 20:08:39.788492 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.788434    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5afb5f9a-b72c-4b88-be69-0e6ea6723ca8" containerName="payload-pre-processing"
Sep 11 20:08:39.792824 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.792805    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"
Sep 11 20:08:39.797187 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.797165    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:08:39.798225 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.798205    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:08:39.821588 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.821563    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"]
Sep 11 20:08:39.867817 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.867784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509a985-44df-4019-9289-096c87594c54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9\" (UID: \"c509a985-44df-4019-9289-096c87594c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"
Sep 11 20:08:39.968809 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.968765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509a985-44df-4019-9289-096c87594c54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9\" (UID: \"c509a985-44df-4019-9289-096c87594c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"
Sep 11 20:08:39.969482 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:39.969462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509a985-44df-4019-9289-096c87594c54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9\" (UID: \"c509a985-44df-4019-9289-096c87594c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"
Sep 11 20:08:40.103052 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:40.102961    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"
Sep 11 20:08:40.405783 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:40.405760    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"]
Sep 11 20:08:40.406404 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:08:40.406373    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc509a985_44df_4019_9289_096c87594c54.slice/crio-53e192720d01afd77debdc0c4cc42fb254449fe12157e4248fae5a5fd2e8fe00 WatchSource:0}: Error finding container 53e192720d01afd77debdc0c4cc42fb254449fe12157e4248fae5a5fd2e8fe00: Status 404 returned error can't find the container with id 53e192720d01afd77debdc0c4cc42fb254449fe12157e4248fae5a5fd2e8fe00
Sep 11 20:08:40.552547 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:40.552513    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9" event={"ID":"c509a985-44df-4019-9289-096c87594c54","Type":"ContainerStarted","Data":"53e192720d01afd77debdc0c4cc42fb254449fe12157e4248fae5a5fd2e8fe00"}
Sep 11 20:08:41.558910 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:41.558879    2571 generic.go:356] "Generic (PLEG): container finished" podID="c509a985-44df-4019-9289-096c87594c54" containerID="2e39abd3b7dcdb1681ae307d5ea6b408b3ac49725b4b7d976bad30924e0ec861" exitCode=0
Sep 11 20:08:41.559324 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:41.558982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9" event={"ID":"c509a985-44df-4019-9289-096c87594c54","Type":"ContainerDied","Data":"2e39abd3b7dcdb1681ae307d5ea6b408b3ac49725b4b7d976bad30924e0ec861"}
Sep 11 20:08:42.687293 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:42.687269    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"
Sep 11 20:08:42.792727 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:42.792694    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509a985-44df-4019-9289-096c87594c54-maas-api-service-ca\") pod \"c509a985-44df-4019-9289-096c87594c54\" (UID: \"c509a985-44df-4019-9289-096c87594c54\") "
Sep 11 20:08:42.793056 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:42.793029    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c509a985-44df-4019-9289-096c87594c54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c509a985-44df-4019-9289-096c87594c54" (UID: "c509a985-44df-4019-9289-096c87594c54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:08:42.894013 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:42.893985    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c509a985-44df-4019-9289-096c87594c54-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:43.571009 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:43.570974    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9" event={"ID":"c509a985-44df-4019-9289-096c87594c54","Type":"ContainerDied","Data":"53e192720d01afd77debdc0c4cc42fb254449fe12157e4248fae5a5fd2e8fe00"}
Sep 11 20:08:43.571240 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:43.571030    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53e192720d01afd77debdc0c4cc42fb254449fe12157e4248fae5a5fd2e8fe00"
Sep 11 20:08:43.571240 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:43.570980    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"
Sep 11 20:08:44.578808 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.578777    2571 generic.go:356] "Generic (PLEG): container finished" podID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerID="7be325ebea0cb2262a84209e91dcdf81dc2080ec64bdf7efb90b07caa31bb625" exitCode=0
Sep 11 20:08:44.579188 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.578835    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" event={"ID":"9adde3c4-cdb6-4b71-8606-39eb008b1469","Type":"ContainerDied","Data":"7be325ebea0cb2262a84209e91dcdf81dc2080ec64bdf7efb90b07caa31bb625"}
Sep 11 20:08:44.579188 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.578884    2571 scope.go:117] "RemoveContainer" containerID="eee7b9c0e7463a1955126aae660acb7bc5917e5e68339eb79ebd7427b6c9b2b5"
Sep 11 20:08:44.658576 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.658552    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:08:44.809856 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.809759    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-envoy\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.809856 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.809839    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-certs\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.810089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.809868    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-credential-socket\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.810089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.809891    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-data\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.810089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.809917    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5rb8\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-kube-api-access-d5rb8\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.810089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.809938    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-token\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.810089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810006    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adde3c4-cdb6-4b71-8606-39eb008b1469-istiod-ca-cert\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.810089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810030    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-podinfo\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.810429 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810153    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:44.810429 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810239    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-socket\") pod \"9adde3c4-cdb6-4b71-8606-39eb008b1469\" (UID: \"9adde3c4-cdb6-4b71-8606-39eb008b1469\") "
Sep 11 20:08:44.810429 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810269    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-data" (OuterVolumeSpecName: "istio-data") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:44.810429 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810279    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:44.810602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810488    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:44.810602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810506    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-credential-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:44.810602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810507    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:44.810602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810520    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-data\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:44.810602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.810524    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adde3c4-cdb6-4b71-8606-39eb008b1469-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:08:44.812164 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.812143    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 20:08:44.812353 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.812332    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:08:44.812602 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.812577    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-kube-api-access-d5rb8" (OuterVolumeSpecName: "kube-api-access-d5rb8") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "kube-api-access-d5rb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:08:44.812666 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.812601    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-token" (OuterVolumeSpecName: "istio-token") pod "9adde3c4-cdb6-4b71-8606-39eb008b1469" (UID: "9adde3c4-cdb6-4b71-8606-39eb008b1469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:08:44.910951 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.910917    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-podinfo\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:44.910951 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.910945    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-workload-socket\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:44.910951 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.910954    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-envoy\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:44.911206 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.910963    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5rb8\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-kube-api-access-d5rb8\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:44.911206 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.910972    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adde3c4-cdb6-4b71-8606-39eb008b1469-istio-token\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:44.911206 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:44.910981    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adde3c4-cdb6-4b71-8606-39eb008b1469-istiod-ca-cert\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:08:45.585579 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:45.585547    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"
Sep 11 20:08:45.585579 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:45.585564    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn" event={"ID":"9adde3c4-cdb6-4b71-8606-39eb008b1469","Type":"ContainerDied","Data":"e9f31421375602fc68db3d60eef9920fb13a800420915a6e93d7baacbbbc23c9"}
Sep 11 20:08:45.586076 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:45.585619    2571 scope.go:117] "RemoveContainer" containerID="7be325ebea0cb2262a84209e91dcdf81dc2080ec64bdf7efb90b07caa31bb625"
Sep 11 20:08:45.609210 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:45.609171    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"]
Sep 11 20:08:45.613143 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:45.613121    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df1a8a-openshift-default-5fc77cd8f4-bkjtn"]
Sep 11 20:08:45.873224 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:45.873196    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" path="/var/lib/kubelet/pods/9adde3c4-cdb6-4b71-8606-39eb008b1469/volumes"
Sep 11 20:08:50.521880 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:50.521838    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"]
Sep 11 20:08:50.524028 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:50.524004    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-57d97d-7kpd9"]
Sep 11 20:08:51.873294 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:51.873260    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c509a985-44df-4019-9289-096c87594c54" path="/var/lib/kubelet/pods/c509a985-44df-4019-9289-096c87594c54/volumes"
Sep 11 20:08:53.549398 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:53.549367    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:08:57.379702 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.379658    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"]
Sep 11 20:08:57.380201 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380122    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy"
Sep 11 20:08:57.380201 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380141    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="03884f0e-1142-4530-a781-a4dfebb30804" containerName="istio-proxy"
Sep 11 20:08:57.380201 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380161    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c509a985-44df-4019-9289-096c87594c54" containerName="revoke-keys"
Sep 11 20:08:57.380201 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380169    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c509a985-44df-4019-9289-096c87594c54" containerName="revoke-keys"
Sep 11 20:08:57.380201 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380191    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy"
Sep 11 20:08:57.380201 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380199    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy"
Sep 11 20:08:57.380576 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380213    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy"
Sep 11 20:08:57.380576 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380222    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy"
Sep 11 20:08:57.380576 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380296    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy"
Sep 11 20:08:57.380576 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.380310    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c509a985-44df-4019-9289-096c87594c54" containerName="revoke-keys"
Sep 11 20:08:57.384872 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.384850    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"
Sep 11 20:08:57.387608 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.387586    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9dhwn\""
Sep 11 20:08:57.387903 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.387881    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:08:57.391456 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.391433    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"]
Sep 11 20:08:57.508416 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.508384    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95ec082-8d40-4606-9108-929490077185-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6\" (UID: \"e95ec082-8d40-4606-9108-929490077185\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"
Sep 11 20:08:57.609226 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.609199    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95ec082-8d40-4606-9108-929490077185-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6\" (UID: \"e95ec082-8d40-4606-9108-929490077185\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"
Sep 11 20:08:57.609827 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.609801    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95ec082-8d40-4606-9108-929490077185-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6\" (UID: \"e95ec082-8d40-4606-9108-929490077185\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"
Sep 11 20:08:57.695958 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.695896    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"
Sep 11 20:08:57.970862 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:57.970836    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"]
Sep 11 20:08:57.973375 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:08:57.973341    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95ec082_8d40_4606_9108_929490077185.slice/crio-4754d563924381ad3c7b0a61d1f47e0cba07cc64392b78b70efeb920eaca440b WatchSource:0}: Error finding container 4754d563924381ad3c7b0a61d1f47e0cba07cc64392b78b70efeb920eaca440b: Status 404 returned error can't find the container with id 4754d563924381ad3c7b0a61d1f47e0cba07cc64392b78b70efeb920eaca440b
Sep 11 20:08:58.650720 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:58.650688    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6" event={"ID":"e95ec082-8d40-4606-9108-929490077185","Type":"ContainerStarted","Data":"7d7d120febc88abccd6aff3e60fa9b9574a3f398518f752418aa04c305ff8ea9"}
Sep 11 20:08:58.651076 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:58.650739    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6" event={"ID":"e95ec082-8d40-4606-9108-929490077185","Type":"ContainerStarted","Data":"4754d563924381ad3c7b0a61d1f47e0cba07cc64392b78b70efeb920eaca440b"}
Sep 11 20:08:58.667255 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:58.667209    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6" podStartSLOduration=1.667195853 podStartE2EDuration="1.667195853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:58.665415552 +0000 UTC m=+2375.341352459" watchObservedRunningTime="2026-09-11 20:08:58.667195853 +0000 UTC m=+2375.343132760"
Sep 11 20:08:59.657011 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:59.656967    2571 generic.go:356] "Generic (PLEG): container finished" podID="e95ec082-8d40-4606-9108-929490077185" containerID="7d7d120febc88abccd6aff3e60fa9b9574a3f398518f752418aa04c305ff8ea9" exitCode=0
Sep 11 20:08:59.657428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:08:59.657055    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6" event={"ID":"e95ec082-8d40-4606-9108-929490077185","Type":"ContainerDied","Data":"7d7d120febc88abccd6aff3e60fa9b9574a3f398518f752418aa04c305ff8ea9"}
Sep 11 20:09:00.782179 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:00.782155    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"
Sep 11 20:09:00.835553 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:00.835521    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95ec082-8d40-4606-9108-929490077185-maas-api-service-ca\") pod \"e95ec082-8d40-4606-9108-929490077185\" (UID: \"e95ec082-8d40-4606-9108-929490077185\") "
Sep 11 20:09:00.835857 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:00.835835    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95ec082-8d40-4606-9108-929490077185-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e95ec082-8d40-4606-9108-929490077185" (UID: "e95ec082-8d40-4606-9108-929490077185"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:09:00.936088 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:00.936006    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95ec082-8d40-4606-9108-929490077185-maas-api-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:09:01.672952 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:01.672920    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"
Sep 11 20:09:01.672952 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:01.672933    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6" event={"ID":"e95ec082-8d40-4606-9108-929490077185","Type":"ContainerDied","Data":"4754d563924381ad3c7b0a61d1f47e0cba07cc64392b78b70efeb920eaca440b"}
Sep 11 20:09:01.673215 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:01.673001    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4754d563924381ad3c7b0a61d1f47e0cba07cc64392b78b70efeb920eaca440b"
Sep 11 20:09:07.984798 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:07.984767    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"]
Sep 11 20:09:07.989068 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:07.989037    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8a4eb7-29819280-gmb7m"]
Sep 11 20:09:08.020809 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.020781    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"]
Sep 11 20:09:08.021113 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.021072    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l" podUID="978af7f4-f1e8-427f-9fe0-777d92eb0ba4" containerName="payload-processing" containerID="cri-o://e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d" gracePeriod=30
Sep 11 20:09:08.198784 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.198754    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"]
Sep 11 20:09:08.201652 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.201624    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8a4eb7-6qsf6"]
Sep 11 20:09:08.259997 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.259970    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 20:09:08.289484 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.289451    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j92j4\" (UniqueName: \"kubernetes.io/projected/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-kube-api-access-j92j4\") pod \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\" (UID: \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\") "
Sep 11 20:09:08.289597 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.289575    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-plugins-config-volume\") pod \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\" (UID: \"978af7f4-f1e8-427f-9fe0-777d92eb0ba4\") "
Sep 11 20:09:08.289933 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.289907    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "978af7f4-f1e8-427f-9fe0-777d92eb0ba4" (UID: "978af7f4-f1e8-427f-9fe0-777d92eb0ba4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:09:08.291460 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.291435    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-kube-api-access-j92j4" (OuterVolumeSpecName: "kube-api-access-j92j4") pod "978af7f4-f1e8-427f-9fe0-777d92eb0ba4" (UID: "978af7f4-f1e8-427f-9fe0-777d92eb0ba4"). InnerVolumeSpecName "kube-api-access-j92j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:09:08.390547 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.390514    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-plugins-config-volume\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:09:08.390547 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.390545    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92j4\" (UniqueName: \"kubernetes.io/projected/978af7f4-f1e8-427f-9fe0-777d92eb0ba4-kube-api-access-j92j4\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:09:08.708020 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.707988    2571 generic.go:356] "Generic (PLEG): container finished" podID="978af7f4-f1e8-427f-9fe0-777d92eb0ba4" containerID="e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d" exitCode=0
Sep 11 20:09:08.708185 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.708028    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"
Sep 11 20:09:08.708185 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.708078    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l" event={"ID":"978af7f4-f1e8-427f-9fe0-777d92eb0ba4","Type":"ContainerDied","Data":"e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d"}
Sep 11 20:09:08.708185 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.708152    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l" event={"ID":"978af7f4-f1e8-427f-9fe0-777d92eb0ba4","Type":"ContainerDied","Data":"6f5dae2d21419295af62bc6a385ea8e9769bccbe673d6a2033044dcee87150dd"}
Sep 11 20:09:08.708185 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.708176    2571 scope.go:117] "RemoveContainer" containerID="e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d"
Sep 11 20:09:08.716700 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.716683    2571 scope.go:117] "RemoveContainer" containerID="e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d"
Sep 11 20:09:08.716971 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:09:08.716954    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d\": container with ID starting with e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d not found: ID does not exist" containerID="e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d"
Sep 11 20:09:08.717016 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.716979    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d"} err="failed to get container status \"e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d\": rpc error: code = NotFound desc = could not find container \"e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d\": container with ID starting with e612065c29e666727539589b1645c603551b762997d4faaf1849b25d3680ed0d not found: ID does not exist"
Sep 11 20:09:08.729275 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.729253    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"]
Sep 11 20:09:08.732876 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:08.732851    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8a4eb7-66bf979995-x6s6l"]
Sep 11 20:09:09.872498 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:09.872463    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978af7f4-f1e8-427f-9fe0-777d92eb0ba4" path="/var/lib/kubelet/pods/978af7f4-f1e8-427f-9fe0-777d92eb0ba4/volumes"
Sep 11 20:09:09.872861 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:09.872765    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce813680-4d39-43ac-90b7-335b7591ea84" path="/var/lib/kubelet/pods/ce813680-4d39-43ac-90b7-335b7591ea84/volumes"
Sep 11 20:09:09.873015 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:09.873002    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e95ec082-8d40-4606-9108-929490077185" path="/var/lib/kubelet/pods/e95ec082-8d40-4606-9108-929490077185/volumes"
Sep 11 20:09:10.841967 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:10.841927    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:09:19.850948 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:19.850913    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:09:24.325993 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:09:24.325962    2571 scope.go:117] "RemoveContainer" containerID="54a4176e686da8c5df46bada8c1082c40541c85b0d12ca4e197d6df4e481b9ec"
Sep 11 20:10:39.843267 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:10:39.843226    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:11:04.046327 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:11:04.046289    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:11:11.944012 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:11:11.943973    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:11:20.149361 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:11:20.149324    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:11:37.840262 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:11:37.840225    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:11:44.351922 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:11:44.351887    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:13:11.202093 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:11.202059    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:13:17.697209 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:17.697169    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:13:23.146647 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.146614    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"]
Sep 11 20:13:23.147229 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.147020    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978af7f4-f1e8-427f-9fe0-777d92eb0ba4" containerName="payload-processing"
Sep 11 20:13:23.147229 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.147039    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="978af7f4-f1e8-427f-9fe0-777d92eb0ba4" containerName="payload-processing"
Sep 11 20:13:23.147229 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.147053    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95ec082-8d40-4606-9108-929490077185" containerName="revoke-keys"
Sep 11 20:13:23.147229 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.147063    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95ec082-8d40-4606-9108-929490077185" containerName="revoke-keys"
Sep 11 20:13:23.147229 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.147171    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95ec082-8d40-4606-9108-929490077185" containerName="revoke-keys"
Sep 11 20:13:23.147229 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.147185    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adde3c4-cdb6-4b71-8606-39eb008b1469" containerName="istio-proxy"
Sep 11 20:13:23.147229 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.147196    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="978af7f4-f1e8-427f-9fe0-777d92eb0ba4" containerName="payload-processing"
Sep 11 20:13:23.150179 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.150158    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.153170 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.153150    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c7b336e9-kserve-self-signed-certs\""
Sep 11 20:13:23.164840 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.164819    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"]
Sep 11 20:13:23.224981 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.224960    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-model-cache\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.225092 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.224995    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c567f095-e535-4eac-8dbf-233e73782218-tls-certs\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.225092 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.225017    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-home\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.225092 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.225042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-tmp-dir\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.225245 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.225124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-dshm\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.225245 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.225165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf5v\" (UniqueName: \"kubernetes.io/projected/c567f095-e535-4eac-8dbf-233e73782218-kube-api-access-mpf5v\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326396 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326369    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf5v\" (UniqueName: \"kubernetes.io/projected/c567f095-e535-4eac-8dbf-233e73782218-kube-api-access-mpf5v\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326499 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-model-cache\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326499 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326439    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c567f095-e535-4eac-8dbf-233e73782218-tls-certs\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326499 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326458    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-home\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326499 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-tmp-dir\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326652 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-dshm\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326831 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326811    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-model-cache\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326883 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326863    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-home\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.326940 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.326922    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-tmp-dir\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.328628 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.328610    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-dshm\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.329047 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.329028    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c567f095-e535-4eac-8dbf-233e73782218-tls-certs\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.335294 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.335272    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf5v\" (UniqueName: \"kubernetes.io/projected/c567f095-e535-4eac-8dbf-233e73782218-kube-api-access-mpf5v\") pod \"e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") " pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.458372 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.458318    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:23.732940 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.732917    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"]
Sep 11 20:13:23.734550 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:13:23.734518    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc567f095_e535_4eac_8dbf_233e73782218.slice/crio-18c4060ab2c730372e4d84029c992e14d98b79d14b600aac318012969b49df0b WatchSource:0}: Error finding container 18c4060ab2c730372e4d84029c992e14d98b79d14b600aac318012969b49df0b: Status 404 returned error can't find the container with id 18c4060ab2c730372e4d84029c992e14d98b79d14b600aac318012969b49df0b
Sep 11 20:13:23.940002 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.939968    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf" event={"ID":"c567f095-e535-4eac-8dbf-233e73782218","Type":"ContainerStarted","Data":"8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34"}
Sep 11 20:13:23.940002 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.940010    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf" event={"ID":"c567f095-e535-4eac-8dbf-233e73782218","Type":"ContainerStarted","Data":"18c4060ab2c730372e4d84029c992e14d98b79d14b600aac318012969b49df0b"}
Sep 11 20:13:23.963591 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:23.963547    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf" podStartSLOduration=0.963533377 podStartE2EDuration="963.533377ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:23.96126447 +0000 UTC m=+2640.637201379" watchObservedRunningTime="2026-09-11 20:13:23.963533377 +0000 UTC m=+2640.639470285"
Sep 11 20:13:24.384640 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:24.384611    2571 scope.go:117] "RemoveContainer" containerID="a70054caf7be6c773141297e4e2ae4e4bd81f003cb0568adca961447fc0dee82"
Sep 11 20:13:26.959687 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:26.959652    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:13:33.458833 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:33.458799    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:33.459230 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:33.458845    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:33.471389 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:33.471354    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:33.996768 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:33.996733    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:13:42.251421 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.251387    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"]
Sep 11 20:13:42.254418 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.254397    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.257705 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.257679    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c7b336e9-kserve-self-signed-certs\""
Sep 11 20:13:42.265688 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.265667    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"]
Sep 11 20:13:42.379717 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.379688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-tmp-dir\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.379836 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.379743    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-dshm\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.379836 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.379800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-home\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.379836 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.379824    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-model-cache\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.379951 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.379858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310555fe-907c-4b5c-b9a6-9a0465d40300-tls-certs\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.379951 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.379887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kchc8\" (UniqueName: \"kubernetes.io/projected/310555fe-907c-4b5c-b9a6-9a0465d40300-kube-api-access-kchc8\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.480463 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.480434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310555fe-907c-4b5c-b9a6-9a0465d40300-tls-certs\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.480583 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.480472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kchc8\" (UniqueName: \"kubernetes.io/projected/310555fe-907c-4b5c-b9a6-9a0465d40300-kube-api-access-kchc8\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.480583 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.480503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-tmp-dir\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.480583 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.480551    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-dshm\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.480757 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.480741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-home\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.480814 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.480775    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-model-cache\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.480952 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.480929    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-tmp-dir\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.481090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.481071    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-model-cache\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.481164 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.481123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-home\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.482647 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.482627    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-dshm\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.482932 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.482913    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310555fe-907c-4b5c-b9a6-9a0465d40300-tls-certs\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.489790 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.489763    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kchc8\" (UniqueName: \"kubernetes.io/projected/310555fe-907c-4b5c-b9a6-9a0465d40300-kube-api-access-kchc8\") pod \"e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") " pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.563824 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.563755    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:42.833243 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:42.833161    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"]
Sep 11 20:13:42.837274 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:13:42.837244    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod310555fe_907c_4b5c_b9a6_9a0465d40300.slice/crio-7d0c4593a5026c20d77a04a190240968f6922a9c7a61bc6196ed95db5e137be4 WatchSource:0}: Error finding container 7d0c4593a5026c20d77a04a190240968f6922a9c7a61bc6196ed95db5e137be4: Status 404 returned error can't find the container with id 7d0c4593a5026c20d77a04a190240968f6922a9c7a61bc6196ed95db5e137be4
Sep 11 20:13:43.027626 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:43.027593    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c" event={"ID":"310555fe-907c-4b5c-b9a6-9a0465d40300","Type":"ContainerStarted","Data":"3ebc1a1e59c7d7f9ae3098f8252ae397fc2d62bb507d3b834231159f5fc4067c"}
Sep 11 20:13:43.027626 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:43.027631    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c" event={"ID":"310555fe-907c-4b5c-b9a6-9a0465d40300","Type":"ContainerStarted","Data":"7d0c4593a5026c20d77a04a190240968f6922a9c7a61bc6196ed95db5e137be4"}
Sep 11 20:13:43.057856 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:43.057803    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c" podStartSLOduration=1.057788893 podStartE2EDuration="1.057788893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:43.055878541 +0000 UTC m=+2659.731815449" watchObservedRunningTime="2026-09-11 20:13:43.057788893 +0000 UTC m=+2659.733725826"
Sep 11 20:13:45.352173 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:45.352138    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:13:52.564344 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:52.564309    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:52.564728 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:52.564352    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:52.576541 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:52.576515    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:13:53.085732 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:13:53.085702    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:14:01.795605 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:01.795570    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"]
Sep 11 20:14:03.952254 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:03.952211    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"]
Sep 11 20:14:03.965612 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:03.965585    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:14:11.563963 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:11.563927    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:14:16.814344 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:16.814232    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf" podUID="c567f095-e535-4eac-8dbf-233e73782218" containerName="main" containerID="cri-o://8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34" gracePeriod=45
Sep 11 20:14:17.049467 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.049439    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:14:17.129503 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129480    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-home\") pod \"c567f095-e535-4eac-8dbf-233e73782218\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") "
Sep 11 20:14:17.129651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129529    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c567f095-e535-4eac-8dbf-233e73782218-tls-certs\") pod \"c567f095-e535-4eac-8dbf-233e73782218\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") "
Sep 11 20:14:17.129651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129556    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpf5v\" (UniqueName: \"kubernetes.io/projected/c567f095-e535-4eac-8dbf-233e73782218-kube-api-access-mpf5v\") pod \"c567f095-e535-4eac-8dbf-233e73782218\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") "
Sep 11 20:14:17.129651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129605    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-tmp-dir\") pod \"c567f095-e535-4eac-8dbf-233e73782218\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") "
Sep 11 20:14:17.129651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129630    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-model-cache\") pod \"c567f095-e535-4eac-8dbf-233e73782218\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") "
Sep 11 20:14:17.129858 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129659    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-dshm\") pod \"c567f095-e535-4eac-8dbf-233e73782218\" (UID: \"c567f095-e535-4eac-8dbf-233e73782218\") "
Sep 11 20:14:17.129858 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129783    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-home" (OuterVolumeSpecName: "home") pod "c567f095-e535-4eac-8dbf-233e73782218" (UID: "c567f095-e535-4eac-8dbf-233e73782218"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:14:17.129972 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129877    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-model-cache" (OuterVolumeSpecName: "model-cache") pod "c567f095-e535-4eac-8dbf-233e73782218" (UID: "c567f095-e535-4eac-8dbf-233e73782218"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:14:17.129972 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129889    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c567f095-e535-4eac-8dbf-233e73782218" (UID: "c567f095-e535-4eac-8dbf-233e73782218"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:14:17.129972 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.129925    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:17.131727 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.131690    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-dshm" (OuterVolumeSpecName: "dshm") pod "c567f095-e535-4eac-8dbf-233e73782218" (UID: "c567f095-e535-4eac-8dbf-233e73782218"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:14:17.131829 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.131778    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c567f095-e535-4eac-8dbf-233e73782218-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c567f095-e535-4eac-8dbf-233e73782218" (UID: "c567f095-e535-4eac-8dbf-233e73782218"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:14:17.131829 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.131790    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c567f095-e535-4eac-8dbf-233e73782218-kube-api-access-mpf5v" (OuterVolumeSpecName: "kube-api-access-mpf5v") pod "c567f095-e535-4eac-8dbf-233e73782218" (UID: "c567f095-e535-4eac-8dbf-233e73782218"). InnerVolumeSpecName "kube-api-access-mpf5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:14:17.191428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.191403    2571 generic.go:356] "Generic (PLEG): container finished" podID="c567f095-e535-4eac-8dbf-233e73782218" containerID="8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34" exitCode=0
Sep 11 20:14:17.191535 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.191442    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"
Sep 11 20:14:17.191535 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.191485    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf" event={"ID":"c567f095-e535-4eac-8dbf-233e73782218","Type":"ContainerDied","Data":"8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34"}
Sep 11 20:14:17.191535 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.191526    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf" event={"ID":"c567f095-e535-4eac-8dbf-233e73782218","Type":"ContainerDied","Data":"18c4060ab2c730372e4d84029c992e14d98b79d14b600aac318012969b49df0b"}
Sep 11 20:14:17.191651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.191543    2571 scope.go:117] "RemoveContainer" containerID="8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34"
Sep 11 20:14:17.200433 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.200415    2571 scope.go:117] "RemoveContainer" containerID="8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34"
Sep 11 20:14:17.200705 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:14:17.200686    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34\": container with ID starting with 8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34 not found: ID does not exist" containerID="8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34"
Sep 11 20:14:17.200765 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.200715    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34"} err="failed to get container status \"8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34\": rpc error: code = NotFound desc = could not find container \"8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34\": container with ID starting with 8ea96e190f554911c0b4465ebf63771ad7a4d4ac67415291c64fab0902de1d34 not found: ID does not exist"
Sep 11 20:14:17.214028 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.214004    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"]
Sep 11 20:14:17.218041 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.218022    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c7b336e9-kserve-678bb54495-ps7wf"]
Sep 11 20:14:17.230567 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.230548    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c567f095-e535-4eac-8dbf-233e73782218-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:17.230647 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.230569    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpf5v\" (UniqueName: \"kubernetes.io/projected/c567f095-e535-4eac-8dbf-233e73782218-kube-api-access-mpf5v\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:17.230647 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.230579    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:17.230647 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.230589    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:17.230647 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.230598    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c567f095-e535-4eac-8dbf-233e73782218-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:17.351438 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.351410    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:14:17.872953 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:17.872923    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c567f095-e535-4eac-8dbf-233e73782218" path="/var/lib/kubelet/pods/c567f095-e535-4eac-8dbf-233e73782218/volumes"
Sep 11 20:14:18.972233 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:18.972190    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c" podUID="310555fe-907c-4b5c-b9a6-9a0465d40300" containerName="main" containerID="cri-o://3ebc1a1e59c7d7f9ae3098f8252ae397fc2d62bb507d3b834231159f5fc4067c" gracePeriod=45
Sep 11 20:14:19.202373 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.202346    2571 generic.go:356] "Generic (PLEG): container finished" podID="310555fe-907c-4b5c-b9a6-9a0465d40300" containerID="3ebc1a1e59c7d7f9ae3098f8252ae397fc2d62bb507d3b834231159f5fc4067c" exitCode=0
Sep 11 20:14:19.202514 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.202413    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c" event={"ID":"310555fe-907c-4b5c-b9a6-9a0465d40300","Type":"ContainerDied","Data":"3ebc1a1e59c7d7f9ae3098f8252ae397fc2d62bb507d3b834231159f5fc4067c"}
Sep 11 20:14:19.202514 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.202459    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c" event={"ID":"310555fe-907c-4b5c-b9a6-9a0465d40300","Type":"ContainerDied","Data":"7d0c4593a5026c20d77a04a190240968f6922a9c7a61bc6196ed95db5e137be4"}
Sep 11 20:14:19.202514 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.202475    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d0c4593a5026c20d77a04a190240968f6922a9c7a61bc6196ed95db5e137be4"
Sep 11 20:14:19.213255 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.213233    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:14:19.245172 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245085    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-model-cache\") pod \"310555fe-907c-4b5c-b9a6-9a0465d40300\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") "
Sep 11 20:14:19.245280 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245175    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-tmp-dir\") pod \"310555fe-907c-4b5c-b9a6-9a0465d40300\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") "
Sep 11 20:14:19.245280 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245237    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-dshm\") pod \"310555fe-907c-4b5c-b9a6-9a0465d40300\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") "
Sep 11 20:14:19.245280 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245269    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kchc8\" (UniqueName: \"kubernetes.io/projected/310555fe-907c-4b5c-b9a6-9a0465d40300-kube-api-access-kchc8\") pod \"310555fe-907c-4b5c-b9a6-9a0465d40300\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") "
Sep 11 20:14:19.245443 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245302    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-home\") pod \"310555fe-907c-4b5c-b9a6-9a0465d40300\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") "
Sep 11 20:14:19.245443 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245329    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310555fe-907c-4b5c-b9a6-9a0465d40300-tls-certs\") pod \"310555fe-907c-4b5c-b9a6-9a0465d40300\" (UID: \"310555fe-907c-4b5c-b9a6-9a0465d40300\") "
Sep 11 20:14:19.245443 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245374    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-model-cache" (OuterVolumeSpecName: "model-cache") pod "310555fe-907c-4b5c-b9a6-9a0465d40300" (UID: "310555fe-907c-4b5c-b9a6-9a0465d40300"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:14:19.245443 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245400    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "310555fe-907c-4b5c-b9a6-9a0465d40300" (UID: "310555fe-907c-4b5c-b9a6-9a0465d40300"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:14:19.245598 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245537    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-home" (OuterVolumeSpecName: "home") pod "310555fe-907c-4b5c-b9a6-9a0465d40300" (UID: "310555fe-907c-4b5c-b9a6-9a0465d40300"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:14:19.245598 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245573    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-model-cache\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:19.245676 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.245613    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-tmp-dir\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:19.247387 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.247355    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310555fe-907c-4b5c-b9a6-9a0465d40300-kube-api-access-kchc8" (OuterVolumeSpecName: "kube-api-access-kchc8") pod "310555fe-907c-4b5c-b9a6-9a0465d40300" (UID: "310555fe-907c-4b5c-b9a6-9a0465d40300"). InnerVolumeSpecName "kube-api-access-kchc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:14:19.247480 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.247396    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-dshm" (OuterVolumeSpecName: "dshm") pod "310555fe-907c-4b5c-b9a6-9a0465d40300" (UID: "310555fe-907c-4b5c-b9a6-9a0465d40300"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:14:19.247480 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.247396    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310555fe-907c-4b5c-b9a6-9a0465d40300-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "310555fe-907c-4b5c-b9a6-9a0465d40300" (UID: "310555fe-907c-4b5c-b9a6-9a0465d40300"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:14:19.346565 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.346531    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-dshm\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:19.346565 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.346563    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kchc8\" (UniqueName: \"kubernetes.io/projected/310555fe-907c-4b5c-b9a6-9a0465d40300-kube-api-access-kchc8\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:19.346565 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.346573    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/310555fe-907c-4b5c-b9a6-9a0465d40300-home\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:19.346735 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:19.346582    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310555fe-907c-4b5c-b9a6-9a0465d40300-tls-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:20.206868 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:20.206792    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"
Sep 11 20:14:20.223776 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:20.223749    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"]
Sep 11 20:14:20.228675 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:20.228655    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c7b336e9-kserve-5f9bcff587-qk28c"]
Sep 11 20:14:21.872853 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:21.872811    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="310555fe-907c-4b5c-b9a6-9a0465d40300" path="/var/lib/kubelet/pods/310555fe-907c-4b5c-b9a6-9a0465d40300/volumes"
Sep 11 20:14:24.947119 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:24.947063    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:14:25.063678 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.063643    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"]
Sep 11 20:14:25.063923 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.063911    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c567f095-e535-4eac-8dbf-233e73782218" containerName="main"
Sep 11 20:14:25.063964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.063924    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c567f095-e535-4eac-8dbf-233e73782218" containerName="main"
Sep 11 20:14:25.063964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.063936    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310555fe-907c-4b5c-b9a6-9a0465d40300" containerName="main"
Sep 11 20:14:25.063964 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.063941    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="310555fe-907c-4b5c-b9a6-9a0465d40300" containerName="main"
Sep 11 20:14:25.064061 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.063994    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="310555fe-907c-4b5c-b9a6-9a0465d40300" containerName="main"
Sep 11 20:14:25.064061 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.064004    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c567f095-e535-4eac-8dbf-233e73782218" containerName="main"
Sep 11 20:14:25.068433 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.068415    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.071428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.071404    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 20:14:25.072532 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.072513    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bkww\""
Sep 11 20:14:25.072651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.072531    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 20:14:25.072651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.072562    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 20:14:25.072651 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.072589    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:14:25.079274 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.079251    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"]
Sep 11 20:14:25.191192 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.191166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-openshift-service-ca\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.191340 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.191222    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-maas-api-tls\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.191340 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.191281    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6z2d\" (UniqueName: \"kubernetes.io/projected/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-kube-api-access-d6z2d\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.191449 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.191337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-metrics-certs\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.292333 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.292257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-maas-api-tls\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.292333 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.292289    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6z2d\" (UniqueName: \"kubernetes.io/projected/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-kube-api-access-d6z2d\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.292333 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.292310    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-metrics-certs\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.292333 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.292330    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-openshift-service-ca\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.293042 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.293022    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-openshift-service-ca\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.294746 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.294728    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-metrics-certs\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.294814 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.294788    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-maas-api-tls\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.300827 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.300800    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6z2d\" (UniqueName: \"kubernetes.io/projected/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-kube-api-access-d6z2d\") pod \"maas-api-5787449d97-r4dsw\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") " pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.378647 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.378618    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:25.650913 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.650888    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"]
Sep 11 20:14:25.652731 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:14:25.652697    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2cd4de_5fb9_4bbf_af55_99cd37c451a3.slice/crio-b30dbe410565228e1029249e5bf150f85c2fe79d9427d56ea478cc7c0f8ab9b5 WatchSource:0}: Error finding container b30dbe410565228e1029249e5bf150f85c2fe79d9427d56ea478cc7c0f8ab9b5: Status 404 returned error can't find the container with id b30dbe410565228e1029249e5bf150f85c2fe79d9427d56ea478cc7c0f8ab9b5
Sep 11 20:14:25.654164 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:25.654139    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 20:14:26.237597 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:26.237565    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw" event={"ID":"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3","Type":"ContainerStarted","Data":"8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6"}
Sep 11 20:14:26.237983 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:26.237614    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw" event={"ID":"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3","Type":"ContainerStarted","Data":"b30dbe410565228e1029249e5bf150f85c2fe79d9427d56ea478cc7c0f8ab9b5"}
Sep 11 20:14:26.237983 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:26.237644    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:26.255550 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:26.255507    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw" podStartSLOduration=0.831441009 podStartE2EDuration="1.255490726s" totalImagesPullingTime="424.049717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:26.253409905 +0000 UTC m=+2702.929346826" watchObservedRunningTime="2026-09-11 20:14:26.255490726 +0000 UTC m=+2702.931427634"
Sep 11 20:14:32.247357 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.247329    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:32.546687 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.546605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:14:32.665338 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.665308    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"]
Sep 11 20:14:32.668319 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.668303    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.677530 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.677502    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"]
Sep 11 20:14:32.750004 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.749973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpsh\" (UniqueName: \"kubernetes.io/projected/9b1493a7-5c49-4f50-968e-91285f391f17-kube-api-access-nrpsh\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.750004 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.750006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1493a7-5c49-4f50-968e-91285f391f17-openshift-service-ca\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.750201 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.750029    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b1493a7-5c49-4f50-968e-91285f391f17-metrics-certs\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.750201 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.750096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1493a7-5c49-4f50-968e-91285f391f17-maas-api-tls\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.851044 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.850987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1493a7-5c49-4f50-968e-91285f391f17-maas-api-tls\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.851044 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.851025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpsh\" (UniqueName: \"kubernetes.io/projected/9b1493a7-5c49-4f50-968e-91285f391f17-kube-api-access-nrpsh\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.851044 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.851044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1493a7-5c49-4f50-968e-91285f391f17-openshift-service-ca\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.851236 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.851065    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b1493a7-5c49-4f50-968e-91285f391f17-metrics-certs\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.851739 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.851711    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1493a7-5c49-4f50-968e-91285f391f17-openshift-service-ca\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.853363 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.853343    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b1493a7-5c49-4f50-968e-91285f391f17-metrics-certs\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.853452 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.853411    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b1493a7-5c49-4f50-968e-91285f391f17-maas-api-tls\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.859596 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.859577    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpsh\" (UniqueName: \"kubernetes.io/projected/9b1493a7-5c49-4f50-968e-91285f391f17-kube-api-access-nrpsh\") pod \"maas-api-5f698fdf-c2hjq\" (UID: \"9b1493a7-5c49-4f50-968e-91285f391f17\") " pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:32.979009 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:32.978977    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:33.249619 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:33.249594    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"]
Sep 11 20:14:33.251393 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:14:33.251367    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1493a7_5c49_4f50_968e_91285f391f17.slice/crio-13980513daad6f4a5c41c210ecf32aa96fe8a90889b97af4ecc88a695d9d0816 WatchSource:0}: Error finding container 13980513daad6f4a5c41c210ecf32aa96fe8a90889b97af4ecc88a695d9d0816: Status 404 returned error can't find the container with id 13980513daad6f4a5c41c210ecf32aa96fe8a90889b97af4ecc88a695d9d0816
Sep 11 20:14:33.272979 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:33.272954    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq" event={"ID":"9b1493a7-5c49-4f50-968e-91285f391f17","Type":"ContainerStarted","Data":"13980513daad6f4a5c41c210ecf32aa96fe8a90889b97af4ecc88a695d9d0816"}
Sep 11 20:14:34.278623 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:34.278585    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq" event={"ID":"9b1493a7-5c49-4f50-968e-91285f391f17","Type":"ContainerStarted","Data":"cea6435a6f6a97cd0874cbbeac1d96f1c6bbcb9e29ee27e8a247df841164c224"}
Sep 11 20:14:34.279003 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:34.278650    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:34.297255 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:34.297205    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq" podStartSLOduration=1.953790375 podStartE2EDuration="2.297190267s" totalImagesPullingTime="343.399892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:34.29539128 +0000 UTC m=+2710.971328191" watchObservedRunningTime="2026-09-11 20:14:34.297190267 +0000 UTC m=+2710.973127174"
Sep 11 20:14:38.744636 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:38.744599    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:14:40.288829 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:40.288802    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f698fdf-c2hjq"
Sep 11 20:14:40.337687 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:40.337656    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"]
Sep 11 20:14:40.337857 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:40.337822    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw" podUID="6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" containerName="maas-api" containerID="cri-o://8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6" gracePeriod=30
Sep 11 20:14:41.566819 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.566798    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:41.715089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.715004    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-openshift-service-ca\") pod \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") "
Sep 11 20:14:41.715089 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.715080    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-metrics-certs\") pod \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") "
Sep 11 20:14:41.715323 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.715130    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-maas-api-tls\") pod \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") "
Sep 11 20:14:41.715323 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.715176    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6z2d\" (UniqueName: \"kubernetes.io/projected/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-kube-api-access-d6z2d\") pod \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\" (UID: \"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3\") "
Sep 11 20:14:41.715446 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.715420    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" (UID: "6d2cd4de-5fb9-4bbf-af55-99cd37c451a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:14:41.717167 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.717130    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" (UID: "6d2cd4de-5fb9-4bbf-af55-99cd37c451a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:14:41.717290 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.717171    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-kube-api-access-d6z2d" (OuterVolumeSpecName: "kube-api-access-d6z2d") pod "6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" (UID: "6d2cd4de-5fb9-4bbf-af55-99cd37c451a3"). InnerVolumeSpecName "kube-api-access-d6z2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:14:41.717290 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.717194    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" (UID: "6d2cd4de-5fb9-4bbf-af55-99cd37c451a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:14:41.816597 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.816567    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6z2d\" (UniqueName: \"kubernetes.io/projected/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-kube-api-access-d6z2d\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:41.816597 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.816595    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-openshift-service-ca\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:41.816597 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.816605    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-metrics-certs\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:41.816766 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:41.816613    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3-maas-api-tls\") on node \"ip-10-0-136-27.ec2.internal\" DevicePath \"\""
Sep 11 20:14:42.320091 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.320061    2571 generic.go:356] "Generic (PLEG): container finished" podID="6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" containerID="8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6" exitCode=0
Sep 11 20:14:42.320244 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.320171    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"
Sep 11 20:14:42.320291 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.320174    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw" event={"ID":"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3","Type":"ContainerDied","Data":"8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6"}
Sep 11 20:14:42.320327 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.320296    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5787449d97-r4dsw" event={"ID":"6d2cd4de-5fb9-4bbf-af55-99cd37c451a3","Type":"ContainerDied","Data":"b30dbe410565228e1029249e5bf150f85c2fe79d9427d56ea478cc7c0f8ab9b5"}
Sep 11 20:14:42.320361 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.320325    2571 scope.go:117] "RemoveContainer" containerID="8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6"
Sep 11 20:14:42.328702 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.328690    2571 scope.go:117] "RemoveContainer" containerID="8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6"
Sep 11 20:14:42.329030 ip-10-0-136-27 kubenswrapper[2571]: E0911 20:14:42.329010    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6\": container with ID starting with 8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6 not found: ID does not exist" containerID="8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6"
Sep 11 20:14:42.329097 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.329039    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6"} err="failed to get container status \"8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6\": rpc error: code = NotFound desc = could not find container \"8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6\": container with ID starting with 8750ae50a9b222d0c5c5b99bb7045a9af3a85b9fd7a3a8b1ae3f01203ea2cde6 not found: ID does not exist"
Sep 11 20:14:42.339886 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.339865    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"]
Sep 11 20:14:42.344146 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:42.344127    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5787449d97-r4dsw"]
Sep 11 20:14:43.872759 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:43.872726    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" path="/var/lib/kubelet/pods/6d2cd4de-5fb9-4bbf-af55-99cd37c451a3/volumes"
Sep 11 20:14:47.744173 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:47.744137    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:14:53.145962 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:14:53.145913    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tb4n9"]
Sep 11 20:15:01.561200 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:01.561172    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4_bf216ac5-0777-4aaf-a0d5-ddca19b337a7/copy-manifests/0.log"
Sep 11 20:15:01.567742 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:01.567713    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4_bf216ac5-0777-4aaf-a0d5-ddca19b337a7/manager/0.log"
Sep 11 20:15:02.008934 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:02.008900    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5f698fdf-c2hjq_9b1493a7-5c49-4f50-968e-91285f391f17/maas-api/0.log"
Sep 11 20:15:02.226918 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:02.226891    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819265-9wrsb_007ba46e-aa9e-4dfc-9ca2-6c77bf626bb9/cleanup/0.log"
Sep 11 20:15:03.866411 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:03.866380    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tb4n9_56d7065b-4d2b-4705-98eb-30a3ad5e0ee6/limitador/0.log"
Sep 11 20:15:04.539059 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:04.539033    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d55d58b74-dpn26_f176b659-74c0-4e11-8d2b-2b78a8cf663b/kube-auth-proxy/0.log"
Sep 11 20:15:05.506027 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:05.505998    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-s8hpw_6db52705-77cf-49c1-bf71-98f8cc5dfe30/main/0.log"
Sep 11 20:15:05.937562 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:05.937531    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-fmc5h_da82de53-777b-471a-b96b-adc1519fc6f8/main/0.log"
Sep 11 20:15:06.041872 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:06.041840    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cgmk7_bc4e3640-c581-4541-a863-e5c00c26c30c/main/0.log"
Sep 11 20:15:14.320162 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:14.320124    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z292n_412e8257-acb6-41f7-80de-b26261cb8fcb/global-pull-secret-syncer/0.log"
Sep 11 20:15:14.402428 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:14.402399    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n22pk_d24c309d-5363-43b8-a580-5e9c3c11de02/konnectivity-agent/0.log"
Sep 11 20:15:14.458752 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:14.458723    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-27.ec2.internal_496e13ce30a424109c88ee9e45a70e85/haproxy/0.log"
Sep 11 20:15:19.358069 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:19.358037    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tb4n9_56d7065b-4d2b-4705-98eb-30a3ad5e0ee6/limitador/0.log"
Sep 11 20:15:21.319866 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:21.319838    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhbbb_5392b55e-78e6-4494-8fbe-f5387f6c4cb3/node-exporter/0.log"
Sep 11 20:15:21.340637 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:21.340612    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhbbb_5392b55e-78e6-4494-8fbe-f5387f6c4cb3/kube-rbac-proxy/0.log"
Sep 11 20:15:21.361286 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:21.361266    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhbbb_5392b55e-78e6-4494-8fbe-f5387f6c4cb3/init-textfile/0.log"
Sep 11 20:15:24.414811 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.414775    2571 scope.go:117] "RemoveContainer" containerID="7d7d120febc88abccd6aff3e60fa9b9574a3f398518f752418aa04c305ff8ea9"
Sep 11 20:15:24.423018 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.422997    2571 scope.go:117] "RemoveContainer" containerID="2e39abd3b7dcdb1681ae307d5ea6b408b3ac49725b4b7d976bad30924e0ec861"
Sep 11 20:15:24.649265 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.649229    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"]
Sep 11 20:15:24.649527 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.649514    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" containerName="maas-api"
Sep 11 20:15:24.649617 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.649528    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" containerName="maas-api"
Sep 11 20:15:24.649617 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.649586    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2cd4de-5fb9-4bbf-af55-99cd37c451a3" containerName="maas-api"
Sep 11 20:15:24.653800 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.653778    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.656487 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.656466    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89mbl\"/\"openshift-service-ca.crt\""
Sep 11 20:15:24.657822 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.657802    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-89mbl\"/\"default-dockercfg-xmf8z\""
Sep 11 20:15:24.657919 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.657836    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89mbl\"/\"kube-root-ca.crt\""
Sep 11 20:15:24.665748 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.665700    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"]
Sep 11 20:15:24.743213 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.743182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-sys\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.743376 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.743226    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-podres\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.743376 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.743248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctr9l\" (UniqueName: \"kubernetes.io/projected/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-kube-api-access-ctr9l\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.743376 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.743309    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-lib-modules\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.743376 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.743346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-proc\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844488 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-podres\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844662 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844492    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctr9l\" (UniqueName: \"kubernetes.io/projected/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-kube-api-access-ctr9l\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844662 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844610    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-lib-modules\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844662 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844623    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-podres\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844662 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844659    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-proc\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844867 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844704    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-proc\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844867 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844748    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-sys\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844867 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844810    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-sys\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.844977 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.844869    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-lib-modules\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.852316 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.852291    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctr9l\" (UniqueName: \"kubernetes.io/projected/2d79b03a-e619-418f-bfbf-b5900f9ca6e9-kube-api-access-ctr9l\") pod \"perf-node-gather-daemonset-kg92x\" (UID: \"2d79b03a-e619-418f-bfbf-b5900f9ca6e9\") " pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:24.964092 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:24.963995    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:25.231232 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:25.231207    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"]
Sep 11 20:15:25.233147 ip-10-0-136-27 kubenswrapper[2571]: W0911 20:15:25.233114    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2d79b03a_e619_418f_bfbf_b5900f9ca6e9.slice/crio-9bc8cf65f02840fd78a072790baaa7ac1c38a9c62c3197227cbf07a96898071c WatchSource:0}: Error finding container 9bc8cf65f02840fd78a072790baaa7ac1c38a9c62c3197227cbf07a96898071c: Status 404 returned error can't find the container with id 9bc8cf65f02840fd78a072790baaa7ac1c38a9c62c3197227cbf07a96898071c
Sep 11 20:15:25.531080 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:25.530988    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x" event={"ID":"2d79b03a-e619-418f-bfbf-b5900f9ca6e9","Type":"ContainerStarted","Data":"ebfbba21365b2da68fb3b9db52664c69a562bf3ace1f50106f5cc94afad60c21"}
Sep 11 20:15:25.531080 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:25.531041    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x" event={"ID":"2d79b03a-e619-418f-bfbf-b5900f9ca6e9","Type":"ContainerStarted","Data":"9bc8cf65f02840fd78a072790baaa7ac1c38a9c62c3197227cbf07a96898071c"}
Sep 11 20:15:25.531080 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:25.531062    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:25.547056 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:25.547003    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x" podStartSLOduration=1.5469895120000001 podStartE2EDuration="1.546989512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:15:25.545586719 +0000 UTC m=+2762.221523630" watchObservedRunningTime="2026-09-11 20:15:25.546989512 +0000 UTC m=+2762.222926420"
Sep 11 20:15:26.703232 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:26.703206    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5wg6v_eb80e9a8-e050-4d66-9313-2495201b3238/dns/0.log"
Sep 11 20:15:26.723625 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:26.723590    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5wg6v_eb80e9a8-e050-4d66-9313-2495201b3238/kube-rbac-proxy/0.log"
Sep 11 20:15:26.872024 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:26.871993    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wmckq_25cf9814-dfe6-41da-b125-56e077f8faf8/dns-node-resolver/0.log"
Sep 11 20:15:27.576084 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:27.576044    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xg2gb_c6b1a476-e4f1-4aa3-9252-3cffe7e7e903/node-ca/0.log"
Sep 11 20:15:29.034779 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:29.034750    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d55d58b74-dpn26_f176b659-74c0-4e11-8d2b-2b78a8cf663b/kube-auth-proxy/0.log"
Sep 11 20:15:30.114860 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:30.114822    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tghl6_b5006eaf-06a5-4489-a487-826bf6640a1b/serve-healthcheck-canary/0.log"
Sep 11 20:15:30.676703 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:30.676671    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8fhz_98ef2ecc-f11a-40e7-9982-cb00464b38d4/kube-rbac-proxy/0.log"
Sep 11 20:15:30.702544 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:30.702515    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8fhz_98ef2ecc-f11a-40e7-9982-cb00464b38d4/exporter/0.log"
Sep 11 20:15:30.753090 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:30.753059    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8fhz_98ef2ecc-f11a-40e7-9982-cb00464b38d4/extractor/0.log"
Sep 11 20:15:31.546141 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:31.546093    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-89mbl/perf-node-gather-daemonset-kg92x"
Sep 11 20:15:33.679945 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:33.679911    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4_bf216ac5-0777-4aaf-a0d5-ddca19b337a7/manager/0.log"
Sep 11 20:15:33.696746 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:33.696717    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cccd5967-bp6t4_bf216ac5-0777-4aaf-a0d5-ddca19b337a7/copy-manifests/0.log"
Sep 11 20:15:35.252315 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:35.252285    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f6d84646c-82tkk_71654660-e657-4ba8-9131-0e6ddbfed47f/manager/0.log"
Sep 11 20:15:35.305592 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:35.305559    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hwqf4_1cea7bf3-322d-4ca8-9cf7-50654724afb6/openshift-lws-operator/0.log"
Sep 11 20:15:35.446182 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:35.446159    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d55d58b74-dpn26_f176b659-74c0-4e11-8d2b-2b78a8cf663b/kube-auth-proxy/0.log"
Sep 11 20:15:37.562814 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:37.562786    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d55d58b74-dpn26_f176b659-74c0-4e11-8d2b-2b78a8cf663b/kube-auth-proxy/0.log"
Sep 11 20:15:41.654801 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:41.654761    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6s5tc_6be83e60-4480-4499-9053-044eb1a506f9/kube-multus/0.log"
Sep 11 20:15:41.859063 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:41.859027    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk9z2_fb06fd55-d2a4-46c2-89e8-9247da15014f/kube-multus-additional-cni-plugins/0.log"
Sep 11 20:15:41.878388 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:41.878356    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk9z2_fb06fd55-d2a4-46c2-89e8-9247da15014f/egress-router-binary-copy/0.log"
Sep 11 20:15:41.896838 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:41.896812    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk9z2_fb06fd55-d2a4-46c2-89e8-9247da15014f/cni-plugins/0.log"
Sep 11 20:15:41.916563 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:41.916488    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk9z2_fb06fd55-d2a4-46c2-89e8-9247da15014f/bond-cni-plugin/0.log"
Sep 11 20:15:41.933284 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:41.933262    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk9z2_fb06fd55-d2a4-46c2-89e8-9247da15014f/routeoverride-cni/0.log"
Sep 11 20:15:41.949539 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:41.949513    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk9z2_fb06fd55-d2a4-46c2-89e8-9247da15014f/whereabouts-cni-bincopy/0.log"
Sep 11 20:15:41.965991 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:41.965965    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mk9z2_fb06fd55-d2a4-46c2-89e8-9247da15014f/whereabouts-cni/0.log"
Sep 11 20:15:42.223937 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:42.223852    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mtpxz_d81d2d85-34c6-44f9-a0e4-17ae596fea59/network-metrics-daemon/0.log"
Sep 11 20:15:42.236714 ip-10-0-136-27 kubenswrapper[2571]: I0911 20:15:42.236686    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mtpxz_d81d2d85-34c6-44f9-a0e4-17ae596fea59/kube-rbac-proxy/0.log"